# Cascaded Reinforcement Learning Agents for Large Action Spaces in Autonomous Penetration Testing
## 4 Methodology  
### Algebraic Action Decomposition Scheme  
The main contribution of this paper is to propose an action decomposition strategy to reduce large discrete action spaces into manageable subsets for deep reinforcement learning. Our solution also includes a cooperative training algorithm that stabilises the learning process and supports better convergence properties. As mentioned in the previous section, a large action space U is decomposed into L action component subspaces. However, rather than using different neural network heads to manage these subsets, as proposed by [PERSON] et al. [9], separate DQN agents are used instead. The benefits of this approach are highlighted in the next section. The idea of decomposing a large action space into smaller action component sets is similar to having a tree structure with multiple layers of action selections. Each node in the tree acts like an action component selection node in that it picks a corresponding component from one of its child nodes. Each node has its own range of identifiers. In order to reach a leaf node with a specific action value, the parent node needs to pick the right children to reach a specified leaf. Figure 2, which is an adaptation from a structured parameterised action space in [19], illustrates the idea of having different action selection layers. Inspired from a segment tree data structure, each non-leaf node in the tree contains a range of identifiers that are reachable from that node. All nodes in the tree share the same branching factor, which is considered to be a hyper-parameter of the model. Heuristics are used to calculate the total number of nodes and the number of levels, as shown in Equation (3), where \(b\) is the branching factor. The levels are constructed by filling in the nodes from top to bottom, one level at a time, noting that the final tree should be a complete tree. \[|T|=\log_{b}(|\mathcal{U}|) \tag{3}\]  
Instead of having a different number of agents in each level to represent ranges of integer values, a linear algebraic function shifts the action component identifier values into appropriate ranges. \[a_{out}^{i+1}=f(a_{out}^{i},a^{i+1})=a_{out}^{i}\times\beta^{i+1}+a^{i+1} \tag{4}\]  
In Equation (4), the coefficient \(\beta^{i+1}\) represents the number of action component selection nodes or agents at the level \((i+1)\)th. The value for \(\beta^{i+1}\) shifts the integer values of  
Figure 1: Action space composition. The final action space \(\mathcal{U}\) is constructed by chaining multiple action component’s subspaces \(\mathcal{A}^{i}\). the action components to particular output ranges. This results in having only one agent per level, as shown in Figure 3. This action decomposition scheme shows that, instead of having a single reinforcement learning agent explore and learn an entire action space of \(|\mathcal{U}|\), which can be extremely large, a few smaller reinforcement learning agents can be used to explore and learn in a much smaller version of the action component subspace. The outputs of these smaller agents are then chained together to produce the final action. The complexity of having multiple agents on different levels is similar to traversing a tree in computer science. The number of levels \(L\) is of the order of \(L\approx\log_{b}|\mathcal{U}|\). As a result, only a handful of agents are required to construct an action space even if there are millions of possible actions. To understand the intuition behind the proposed action composition approach, let us structure those small action component sets into a hierarchy, as shown in Figure 2. Each circle in the illustration represents an agent and the arrows are the possible action components. The three smaller sets of action components, \(\mathcal{A}^{1},\mathcal{A}^{2}\) and \(\mathcal{A}^{3}\), are chosen and, for simplicity, each has the same action size of 3. The green highlighted path represents the sequence of action components needed to produce the primitive action 21 which is executable by the environment.