# Combining Supervised and Reinforcement Learning to Build a Generic Defensive Cyber Agent
## 4 Performance Evaluation  
For performance evaluation, the following types of agents are used:  
* A blue agent specifically trained against the \(b\_line\) red agent, referred to as \(BL\);
* A blue agent specifically trained against the \(red\_meander\) red agent, referred to as \(RED\);
* \(BL\) retrained against the \(red\_meander\) red agent, referred to as \(BL\)-\(RED\);
* \(RED\) retrained against the \(b\_line\) red agent, referred to as \(RED\)-\(BL\);
* A proposed framework-based blue agent that uses a Multi-layer Perceptron approach, referred to as \(MLP\).
* A proposed framework-based blue agent that uses a Support Vector Machine approach, referred to as \(SVM\);
* A proposed framework-based blue agent that uses VAE-V, referred to as \(VAE\)-\(V\);
* A hierarchical super agent, a blue agent that first determines the kind of attacking red agent, then launches the specifically trained blue agent for that particular red agent. Variants of the hierarchical blue agent that misclassify the attacking agent by 5%, 10%, 15%, and 20% are referred to as \(HA5\), \(HA10\), \(HA15\), and \(HA20\), respectively.  
The proposed framework integrates RL with supervised learning. To gather data for supervised learning, each trained blue agent is run against the red agent it was trained to oppose, collecting observation-action tuples for further learning. Before we trained each agent, we carefully tuned relevant hyperparameters, Table 5 lists the parameters and the range of values explored, Table 6 shows the optimal hyperparamter values when training against each opponent. For each agent, 10,000 tuples were collected. We employ multi-layer perceptron and support vector machine algorithms for supervised learning, though the framework is adaptable to any supervised learning algorithm. HP optimization for both multi-layer perceptron and support vector machine algorithms was conducted using Bayesian search. For \(SVM\), the HPs considered were \(C\), \(gamma\), and \(kernel\), with optimal values found to be 10.9975, 4.501669 \(\times\) 10\({}^{-6}\), and \(linear\), respectively. For \(MLP\), the HPs considered were the number of hidden layers, activation function, solver, learning rate, and alpha, with optimal values determined to be 10, \(tanh\), \(lbfgs\), \(invscaling\), and 0.01542674, respectively. For VAE-V, the \(ReLU\) activation function is used in each layer, except for the output layer, which uses the \(Softmax\) activation function. The loss function is a combination of categorical cross-entropy and Kullback-Leibler divergence. The latent vector comprises of 18 elements.  
We deployed our approach in CybORG, which features two distinct red agents/attackers. Each blue agent used for performance evaluation is trained for 100,000 steps. For the various performance evaluation scenarios considered here, proximal policy optimization (PPO) is utilized as the RL algorithm, as previous studies have shown its superior performance and generalization ability compared to other RL algorithms like DQN [5]. The agents are trained using the networking scenario illustrated in Figure 3. In each scenario, every trained blue agent is evaluated over 100 episodes against the red agents.  
Our performance evaluation experiments utilize two different episode lengths: 100 steps and 50 steps. By varying the episode length, we aim to assess the trained agents' performance over both shorter and longer periods. This approach helps to highlight how the trained blue agents perform initially and how they adjust their behavior as the scenario progresses. It should also be noted that blue agents do not receive rewards but are penalized at each step. They incur heavier penalties for successful actions of red agents. Therefore, an effective blue agent is one that successfully prevents the red agent's actions, resulting in a total reward closer to zero.