# Combining Supervised and Reinforcement Learning to Build a Generic Defensive Cyber Agent
## 5 Conclusions  
The objective of our work is to train a single, generic blue agent that is independent of any attacking red agent and, to some extent, independent of the network topology as well. The proposed framework combines RL with supervised learning. A variant of the framework uses a custom-built VAE in the supervised learning phase. To explore alternate methods that may result in a generic blue agent, the option of retraining an existing blue agent against another type of red agent is also explored. Our results show that the proposed framework for generic blue agent training does result in a blue agent that possesses generic features in terms of an attacking red agent. Retraining an existing blue agent against another type of red agent tends to result in poor performance. The VAE-based version of the proposed framework demonstrates the best overall performance, i.e., achieves higher total rewards. It also demonstrates strong performance when evaluated in scenarios with network topologies that differ from the one used during agent training. Also, all studied supervised learning methods perform strongly compared to blue agents trained solely using reinforcement learning. In fact, in one scenario (Topology 3), the use of an SVM outperforms the VAE-based version. SVM is also more effective than MLP in scenarios where the network topology is changed. This indicates that using SVM in the supervised learning phase of the proposed framework may result in a blue agent that is more topology-agnostic.  
For future work, we will explore the most suitable approach to achieve generalized strong performance in more depth. In particular, we will apply our approach to the recently launched Cage Challenge 4 [30]. This challenge has a number of interesting features. During both training and evaluation, each time the environment resets, a new scenario is created. The overall network topology in terms of networks, subnets, and physical connectivity remains the some. However, the number of user hosts and servers in each subnet varies, in addition to the number of deployed services on each user host and server. Also, the network experiences multiple operational phases. Different phases support different communication patterns based on pre-configured firewall rules. The reward function varies based on a device's priority during a given phase. Last but not least, Cage Challenge 4 models a multi-agent environment, where, collectively, five blue agents can be trained to defend the complete network, which differs significantly from the single-defender scenario we studied here. As part of our framework, when a more complex red agent is introduced, the RL-based defender agent is trained against the new red strategy. Subsequently, our proposed data collection approach and supervised learning leverage the training of a generic defender capable of addressing a broader range of red agent behaviors. This mechanism allows the framework to generalize across increasingly complex adversarial strategies, supporting scalability to larger and more diverse environments.  
This research received no external funding.  
Data Availability Statement: Data is contained within the article.