# Combining Supervised and Reinforcement Learning to Build a Generic Defensive Cyber Agent
## 2 Background and Related Work
### Reinforcement Learning Environments for Autonomous Cyber Operations  
There are several RL training environments available for the training of agents in ACOs, such as those described in [20; 21; 22; 23]. However, this work utilizes CybORG [24], which was released as part of CAGE Challenge 2. The remainder of this section focuses on CybORG. CybORG: An Autonomous Cyber Operations Research Gym  
Inspired by OpenAI Gym [25], CybORG [24] provides tools that facilitate the application of machine learning in ACOs. More specifically, it provides appropriate tools and environments to train decision-making models using RL in adversarial scenarios such as ACOs. A typical learning environment in CybORG consists of a computer network that is further partitioned into different subnets. A computer network in the learning environment contains the following: different types of hosts and servers, routers, switches, and firewalls. One such scenario in CybORG is shown in Figure 3. Apart from the listed networking entities, CybORG also provides the following teams: attackers and defenders. The attacker team tries to attack a network and disrupt its operation, whereas the defender team tries to defend the network against the attacker team. The attacker and defender teams are also called red and blue, teams respectively. CybORG also provides different types of agents for the red team. Furthermore, it also comes with a green agent, whose task is to generate a network's users normal traffic. CybORG comes with two distinct red agents implemented: (i) _red_meander_ and (ii) _b_line_. _b_line_ is also called a baseline agent. Here, functionalities of red and blue agents are explained with reference to Figure 3. As shown in the figure, the network consists of a user subnet, an enterprise subnet, and an operational subnet. The operational subnet stores information about logistic servers and key manufacturing. Typically, a blue agent is activated by a monitoring system, and it executes a service on each host to detect any malicious software. If malicious software is detected, the agent removes it or, if an attacker has well-established itself, the blue agent restores the system to a clean state using a backup  
Figure 3: A networking scenario in CybORG [24]. mechanism. The red agents start the attack by accessing user hosts which are part of subnet 1. Exploitation and privilege escalation mechanisms are used by an attacker agent to access user hosts. Afterwards, servers in the enterprise subnet are exploited by the red agents. Once a server in the enterprise network is exploited, red agents obtain the IP address of the operational server to disrupt its services. The difference between the two red agents is their attack strategy. The _red_meander_ agent explores the network on a subnet-by-subnet basis. In each subnet, it tries to compromise each host, finally reaching the operational server. On the other hand, the _b_line_ agent, using prior knowledge of the network topology, tries to directly reach the operational server. The same method is used by both agents to decide on the exploitation mechanism to use on different hosts/servers. Primarily, as shown in Figure 2, RL uses the state/observation, action, and reward concept to train an agent. Hence, CybORG comes with action spaces, observations, and rewards for red and blue agents. Tables 1 and 2 list some of the actions that red and blue agents can perform, respectively. A blue agent is able to maintain information about all hosts and servers in a network as part of its observation space. The blue agent's observations about each host consist of the following: (i) interface, (ii) processes, (iii) sessions, (iv) system information, and (v) user information. Interface maintains information about the IP address, subnet, and interface