## 4 Applications, Results, and Discussion
### Performance Indices  
The performance indices quantitatively measure the prediction accuracy and select the most appropriate model after adjusting the model parameters or model formulation to reduce errors in prediction. In comparing the prediction performance for the various parameters, the model with the least errors was considered as the most accurate model. Such errors were determined by comparing the observed and predicted data.  
However, because various performance measures have different prediction errors (e.g., overall error, special point error, and percentage error), the appropriate performance indices must be selected depending on the characteristics of the applied dataset (binary data or real number data). The dataset used in this study is expressed as a binary code (i.e., normal and abnormal condition). Detection performance should be evaluated based on the ability to detect all attacks without raising false alarms.  
Therefore, this study has adopted the classification performance measure approach [51] to correctly classify the state of the WDSs under cyber-attack. The classification performance measure approach makes use of a metric referred to as the sensitivity or true positive rate (_TPR_), which is defined as the ratio of the number of time steps correctly classified as \"under-attack\" to the total number of time steps during which the system is under-attack.  
The metric is composed of four variables, namely, the number of true positives (_TP_), number of false positives (_FP_), number of true negatives (_TN_) and number of false negatives (_FN_). These values are then used to calculate: the precision, also known as the positive predictor value (_PPV_); specificity, also known as the true negative rate (_TNR_); and recall, also known as _TPR_. The equations for these variables are given as follows:  
\[\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} \tag{1}\]  
\[\text{Positive predictor value }(PPV)=\frac{TP}{TP+FP} \tag{2}\]  
\[\text{True Positive rate }(TPR)=\frac{TP}{TP+FN} \tag{3}\]  
\[\text{True Negative rate }(TNR)=\frac{TN}{TN+FP} \tag{4}\]  
To facilitate the comparison among the reported algorithms, this study combined _PPV_ and _TPR_ into the F1 score, which gives the same importance to both metrics. The F1 score is defined as the harmonic average of the two metrics, and is expressed in the following equation:  
\[\text{F1}=2\times\frac{PPV\times TPR}{PPV+TPR} \tag{5}\]Moreover, the time-to-detection (\(TTD\)) value is the difference between the time when an attack starts and when it is first flagged as an under-attack event. It is used to calculate the score of time-to-detection (\(S_{TTD}\)) for effective comparison of all detection models under different attack scenarios. These variables are defined in the following equations:  
\[\text{Time to detection }(TTD)=\text{ t}_{d}-\text{t}_{o}(0\leq TTD\leq\Delta t) \tag{6}\] \[\text{Score of time to detection }(S_{TTD})=1-\frac{1}{n_{a}}\sum_{i=1}^{n_{a}} \frac{TTD_{i}}{\Delta t_{i}} \tag{7}\]  
where \(\Delta t\) is the total duration of the attack, \(n_{a}\) is the number of attacks contained in a dataset, \(TTD_{i}\) is the TTD relative to the \(i\)th attack, and \(\Delta t_{i}\) is the corresponding duration. The overall performance score \(S\) is used to rank the algorithms given as follows:  
\[S=\gamma\times S_{TTD}+(1-\gamma)\times S_{CLF} \tag{8}\]  
where \(S_{CLF}\) is the score of classification performance and is calculated by \(S_{CLF}\) = (\(TPR\) + \(TNR\))/2, and the weight factor for detection importance of time detection or classification performance is \(\gamma\) (set to 0.5 in this study). All metrics described in this section vary between 0 and 1, with 0 representing the poorest performance and 1 representing perfect accuracy.