# Combining Supervised and Reinforcement Learning to Build a Generic Defensive Cyber Agent
## 4 Performance Evaluation
### Results Based on Training Topology  
Figure 7 displays the mean total reward, along with the 95% confidence interval (CI), for different blue agents against _b_line_ and _red_meander_ in 100-step episodes. When the network is attacked by _b_line_, agents trained using the proposed framework show better mean rewards compared to other agents, except BL. Notably, our VAE-V agent achieves the highest mean reward--even better than that of BL, the agent specifically trained for _b_line_. However, this difference is not statistically significant. The REDagent trained specifically for _red_meander_ performs poorly against _b_line_, and this difference is statistically significant. Similarly, BL-RED and RED-BL perform worse against _b_line_. BL-RED performs worse than RED-BL because RED-BL is an agent retrained for _b_line_ after having initially been trained for _red_meander_. Consequently, it demonstrates a better mean reward compared to BL-RED, which was originally trained for _b_line_, then retrained for _red_meander_. This indicates that the agent forgets features of the _b_line_ red agent while retraining. All HA agents show lower mean rewards compared to BL, but the differences in performance are not statistically significant. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Hyper-Parameter** & **b\_Line Red Agent** & **Red Meander Agent** \\ \hline Batch Size & 8 & 256 \\ \hline No. of Steps & 128 & 1024 \\ \hline Gamma & 0.9 & 0.95 \\ \hline Learning Rate & 0.00018937 & 0.00373109 \\ \hline Entropy Coefficient & 1.032461073 \(\times 10^{-5}\) & 0.017615274 \\ \hline Clip Range & 0.3 & 0.3 \\ \hline No. of Epochs & 10 & 10 \\ \hline GAE Lambda & 0.8 & 0.8 \\ \hline Max Gradient Norm & 0.5 & 5 \\ \hline VF Coefficient & 0.09187 & 0.73582 \\ \hline Activation Function & relu & tanh \\ \hline Neural Network Architecture & pi = [256, 256], vf = [256, 256] & pi = [64, 64], vf = [64, 64] \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Tuned hyper-parameter values for blue agents. \begin{table}
\begin{tabular}{l l} \hline \hline
**Hyper-Parameter** & **Range of Values** \\ \hline Batch Size & [8, 16, 32, 64, 128, 256, 512] \\ \hline No. of Steps & [8, 16, 32, 64, 128, 256, 512, 1024, 2048] \\ \hline Gamma & [0.9, 0.95, 0.98, 0.99, 0.995, 0.999, 0.9999] \\ \hline Learning Rate & log range (\(1\times 10^{-5}\), 1) \\ \hline Entropy Coefficient & log range (0.0000001, 0.1) \\ \hline Clip Range & [0.1, 0.2, 0.3, 0.4] \\ \hline No. of Epochs & [1, 5, 10, 20] \\ \hline GAE Lambda & [0.8, 0.9, 0.92, 0.95, 0.98, 0.99, 1.0] \\ \hline Max Gradient Norm & [0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 2, 5] \\ \hline VF Coefficient & [0, 1] \\ \hline Activation Function & [tanh, relu, elu, leaky relu] \\ \hline Neural Network Architecture & [(pi = [64, 64], vf = [64, 64]), (pi = [256, 256], vf = [256, 256])] \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Hyper-parameters and value ranges. For agents' performance against (_red_meander_), the RED agent achieves the best mean reward, as it is specifically trained for _red_meander_. VAE-V and MLP agents, trained using the proposed framework, show similar performance to RED. BL, BL-RED, and RED-BL perform worse compared to RED, with the difference being statistically significant. Notably, RED-BL exhibits a lower mean reward compared to BL-RED, highlighting that retraining of an agent leads to the forgetting of features of the red agent it was originally trained against. HA agents also perform worse compared to RED, and as the error rate in selecting the correct blue agent increases, the performance difference becomes statistically significant between HAs and RED. VAE-V shows a better mean reward compared to HAs, and MLP also demonstrates a better mean reward compared to some HAs. Table 7 shows the total mean reward demonstrated by different agents considering both red agents. MLP and VAE-V exhibit superior total mean rewards, with SVM also outperforming most other agents. The overall best performance is achieved by VAE-V with a score of \(-110.90\), highlighted in bold.