# Combining Supervised and Reinforcement Learning to Build a Generic Defensive Cyber Agent
## 4 Performance Evaluation
### Varying the Network Topology
#### 4.2.1 Topology 2 Results  
Figure 9 depicts the network topology utilized for assessing the efficacy of various blue agent training methods. Henceforth, this topology will be referred to as \"Topology 2\". Compared to the original topology used to train blue agents, the difference here is that _UserHost3_ and _UserHost4_ have been moved from _subnet1_ to _subnet2_. This change should not significantly affect the existing red agents, _b_line_ and _red_meander_, for the following reasons: (i) _b_line_ knows the previous network topology required to reach critical resources and impact the operational server, and moving UserHost3 and UserHost4 does not affect the route to the operational server. (ii) _red_meander_ systematically discovers the network topology, so its behavior should not be significantly impacted. Consequently, no changes were made to the existing red agent scripts. The blue agents trained using the proposed  
Figure 8: Performance comparison using training network topology—50-step episode. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Agent Type** & **BL** & **RED** & **MLP** & **SVM** & **VAE-V** \\ \hline
**Total Mean Reward** & \(-57.4\) & \(-58.83\) & \(-52.39\) & \(-52.92\) & \(-50.4\) \\ \hline
**Agent Type** & **HA5** & **HA10** & **HA15** & **HA20** & \\ \hline
**Total Mean Reward** & \(-50.4\) & \(-52.3\) & \(-54.0\) & \(-55.4\) & \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Total mean reward—50-step episode. framework should be the ones potentially affected by this change, as the number of hosts in is decreased in one subnet and increased in the other. Figure 10 shows the mean reward achieved by different blue agents against the red agents. The results in Figure 10 were collected over 100 episodes, each consisting of 100 steps. Against _b_line_, VAE-V demonstrates the highest mean reward among the evaluated blue agents, followed by SVM, MLP, BL, and HA5. However, the differences in performance are not statistically significant. The RED agent shows the worst performance. Against _red_meander_, VAE-V, again, achieves the best mean reward, followed by SVM, RED, and HA5. Similarly, the performance differences are not statistically significant. BL performs the worst against _red_meander_. Table 9 lists the total mean reward demonstrated by the blue agents considering both red agents, showing that VAE-V achieves the highest total mean reward (indicated by the bold face), followed by SVM, HA5, and MLP. Comparing Tables 7 and 9 reveals the following:  
* The performance of all blue agents trained solely with reinforcement learning (BL, RED, and all HAs) is deteriorated in this modified environment. * The performance of the single blue agent trained with the proposed framework either deteriorates (MLP and VAE-V) or, in the case of SVM, actually demonstrates a slightly improved total mean reward. In summary, combining RL with supervised learning outperforms other defensive agents against the _red\({}_{n}\)eander_ agent, while most defensive agents trained using the proposed framework exhibit comparable performance against the _b\({}_{l}\)ine_ agent when evaluated on a network topology different from the one used during training. Overall, our approach demonstrates the best performance, with the performance of BL, RED, and hierarchical  
Figure 10: Performance comparison using Network Topology 2—100-step episode. Figure 9: Network Topology 2.  
agents showing a more pronounced decline. These findings indicate that the proposed framework for defensive agent training possesses a degree of generalizability to variations in network topology. Figure 11 presents the mean reward demonstrated by the evaluated blue agents for a shorted episode length, i.e., 50 steps. Overall, results are similar to those for 100-step episode, i.e., VAE-V demonstrates the best performance (indicated in bold face), followed by HA5, MLP, and SVM.