# Combining Supervised and Reinforcement Learning to Build a Generic Defensive Cyber Agent
## 3 Framework for Generic Blue Agent Training
### Custom Variational Autoencoder for Generic Blue Agent Training  
The supervised learning component of the proposed framework uses data obtained from interactions between blue agents and red agents. These data are collected during scenarios where blue agents counteract red agents in specific settings, such as a network topology. Even slight changes in network topology can alter the observation vector. Additionally, since the framework's supervised learning part uses data collected from different types of red agents, different observation vectors might be mapped to the same action. This can negatively impact the framework's performance. To address this, we extend the proposed framework by incorporating a method to map observation vectors to a particular distribution, aiming to handle variations in observation vectors and minimize instances where the same observation vector is mapped to different actions. This method is a variation of a variational autoencoder (VAE) and is referred to as the VAE variant, or VAE-V for short.  
In VAE, input data are typically mapped to a normal distribution with the aim of limiting the values that latent variables can take. This process is stochastic in nature, as the values taken by latent variables are sampled from a normal distribution. Once a generic blue agent is trained, it is meant to carry out actions. In our autonomous cyber system, actions are represented as discrete numbers. In a nutshell, in the autonomous cyber system, the generic blue agent expects a vector of observations and outputs a discrete number corresponding to a certain action that needs to be executed. Hence, the problem is exactly analogous to a classification problem. Here, a custom VAE is designed that handles the classification problem in the context of ACOs. The following steps summarize the VAE-V functionality:  
* Each observation tuple is mapped to two latent vectors of dimension \(N\) representing the mean \((\mu)\) and log variance \((\sigma)\) of a standard normal distribution.
* For each latent variable, a point is sampled from a standard normal distribution using the corresponding \(\mu\) and \(\sigma\).
* Latent variables are fed to the decoder to produce an action for a blue agent (the output of this VAE-V is a scalar value).  
Figure 6 shows the designed VAE-V model for generic blue agent training. The encoder part of the model has an input layer \((input\_2)\), two dense layers (_dense_5_ and _dense_6_), and two further dense layers (_z_mean_ and _z_log_var_) to produce a latent vector of dimension 18 (a latent dimension of 18 is chosen after hyper-parameter optimization).The input layer expects a vector of length 52 (in CyORG, the observation tuple consists of 52 items). The decoder part of VAE-V comprises two dense layers (_dense_7_ and _dense_8_) and an output layer (_dense_9_). It should be noted here that the numbers of units in _dense_7_ and _dense_8_ are exactly the same as in _dense_6_ and _dense_5_, respectively. The number of units in the output layer (_dense_9_) is 145 (the total number of actions defined for blue agents in CyORG). As VAE-V is designed for a classification problem in the context of ACOs, the \(softmax\) activation function is used in the output layer to select an action with the highest probability. VAE-V is trained by tuning the latent space dimension.  
Figure 6: Custom VAE model for generic blue agent training.