## 5 Evaluation
### Comparison between One-Class and Binary Classification  
The application of techniques for artificially generating ICS time-series anomalies is clear. When given a dataset containing a small amount of anomalies or even only normal data, augmenting the data with anomalies can help mitigate class imbalance to some extent. Particularly, in the absence of such augmentation techniques, anomaly detection often relies on a one-class classification strategy. Therefore, in this section, the proposed synthetic ICS anomaly is generated to augment the given ICS dataset, and binary classification models are trained and compared with several one-class classification models. Table 3 shows the anomaly detection performance of several one-class classification models (OC) and binary classification models (BIN) for the HAI and SWaT datasets. The one-class classification models adopted are one-class support vector machine (OCSVM), isolation forest (IF), local outlier factor (LOF), and LSTM-based autoencoder (LSTM-AE). The binary classification models adopted are kernel SVM (k-SVM), random forest (RF), k-nearest neighbor (KNN), and LSTM-based binary classifier (LSTM-BIN). The performance indicators used are accuracy (ACC), recall (REC), precision (PRE), F1 (F1 score), and ROC (ROC-AUC). These models are trained to take the ICS time series of length 100 as input and determine whether an anomaly is present. Specifically, any sequence containing non-normal data for at least one timestep is considered an anomaly. Note that no model selection process was conducted to enhance performance, aiming to confirm the differences in performance among models based on the dataset differences. Additionally, the LSTM-AE and LSTM-BIN were designed to have as similar complexity as possible, specifically the same number of parameters. The LSTM-AE consists of an encoder and decoder, each composed of a single LSTM layer with a hidden state size of 256, with a dense layer added at the end of the decoder for reconstruction of the original input sequence. The LSTM-BIN consists of two LSTM layers with a hidden state size of 256, applying a dense layer to the average of the hidden states from the last LSTM layer to classify the given sequence. Both the LSTM-AE and LSTM-BIN were trained for 2048 epochs. Last but not least, all one-class classification models were trained on a training set composed only of normal data, while the binary classification models were trained on a dataset where the original normal data, synthetic normal data (no mutation) by AVRAE, and synthetic anomalies were mixed in a 1:1:1 ratio. In other words, the OC models were trained on the standard dataset, while the BIN models were trained on the dataset augmented with synthetic anomalies. Additionally, the standard training sets for HAI and SWaT do not inherently include anomaly (attack) data. The performance of each model \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**Model**} & \multicolumn{5}{c}{**HAI**} & \multicolumn{5}{c}{**SWaT**} \\ \cline{2-11}  & & **ACC** & **REC** & **PRE** & **F1** & **ROC** & **ACC** & **REC** & **PRE** & **F1** & **ROC** \\ \hline \multirow{4}{*}{**OC**} & **OCSVM** & 0.77 & 0.5 & 0.04 & 0.07 & 0.32 & 0.61 & 0.83 & 0.22 & 0.35 & 0.14 \\  & **IF** & 0.94 & 0.25 & 0.11 & 0.14 & 0.36 & 0.13 & 1.0 & 0.13 & 0.23 & 0.18 \\  & **LOF** & **0.97** & 0.31 & 0.29 & 0.3 & 0.35 & **0.97** & 0.31 & 0.28 & 0.29 & 0.35 \\  & **LSTM-AE** & 0.93 & 0.22 & 0.31 & 0.26 & 0.69 & 0.94 & 0.55 & 0.99 & 0.71 & 0.8 \\ \hline \multirow{4}{*}{**BIN**} & **k-SVM** & 0.37 & 0.79 & 0.06 & 0.11 & 0.54 & 0.86 & 0.71 & 0.47 & 0.57 & **0.85** \\  & **RF** & 0.05 & 1.0 & 0.05 & 0.1 & 0.47 & 0.62 & 0.74 & 0.22 & 0.34 & 0.78 \\  & **KNN** & 0.67 & 0.35 & 0.05 & 0.1 & 0.53 & 0.85 & 0.1 & 0.24 & 0.1 & 0.72 \\  & **LSTM-BIN** & 0.94 & 0.56 & 0.44 & **0.49** & **0.73** & 0.95 & 0.81 & 0.78 & **0.79** & 0.82 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance comparison between one-class and binary classification.