# Cascaded Reinforcement Learning Agents for Large Action Spaces in Autonomous Penetration Testing
## 4 Methodology
### Algebraic Action Decomposition Scheme
The branching factor \(b\) and the number of levels \(L\) are 3 in this example. There are two things worth noting here: First, to reconstruct the original action space \(|\mathcal{U}|=27\), each agent in the hierarchy only needs a small action size of 3. Second, as illustrated in the figure, we need \(1+3+9=13\) agents in total to fully specify the 27 actions. However, we do not have to implement all 13 agents. In fact, we only need three agents, one for each level of the action component selection tree. If the first-level agent picks the action component number 2, and the second-level agent picks the action component number 1, the actual action component identifier created after the second level is \(2*3+1=7\), where 3 is the size of the action space of each agent in the second level. Similarly, the final primitive action identifier is \(7*3+0=21\), where 0 is the action component taken by the corresponding agent at the third level. To generalise this into a compact formula, let us consider the action component signals \(a^{i}\) and \(a^{i+1}\) output by the agents at levels \(i\) and \(i+1\), where \(0<=a^{i}<=|\mathcal{A}^{i}|-1\) and \(0<=a^{i+1}<=|\mathcal{A}^{i+1}|-1\). The action component identifier constructed up to level \(i+1\) is \(a^{i}\times|\mathcal{A}^{i+1}|+a^{i+1}\). The executable primitive action is \(u=a^{L-1}\times|\mathcal{A}^{L}|+a^{L}\), where \(L\) is the number of levels in the action hierarchy. The design of the small individual action component sets \(\mathcal{A}^{1},\dots,\mathcal{A}^{L}\), as well as selecting the value of \(L\), have to be performed manually as no domain knowledge is incorporated into structuring these action component sets. One rule of thumb is to limit the size of each set to be relatively small, say, 10 to 15 actions. The value of \(L\) is then selected accordingly to reconstruct the original action set \(\mathcal{U}\). Each agent in the hierarchy can be implemented using any conventional reinforcement learning algorithm, for example, Dueling DQN [4], which is the core reinforcement learning algorithm we have used in this study. This is our Cascaded Reinforcement Learning Agent architecture, or CRLA for short. An operational diagram of the CRLA architecture is depicted in Figure 3.  
Agents in the hierarchy can be trained in parallel and independently by optimising their own loss functions (Equation (5)). This involves sampling a batch size of \(p\) transitions from the replay buffer, where \(\theta^{i}\) are the parameters of the agent i's network and \(\theta^{-i}\) are the parameters of the target network. The latter parameters are used to stabilise the training, as mentioned in the original DQN paper [1]. \[L^{i}(\theta^{i})=\sum_{j=1}^{p}[(y^{jDDQN}_{j}-Q^{i}(s,a^{i};\theta^{i}))^{2}] \tag{5}\]  
where  
\[y^{j,DDQN}=r_{t}+\gamma\max_{\theta^{i}}Q(s^{\prime},a^{{}^{\prime}i};\theta^ {-i}) \tag{6}\]Figure 3: The operational diagram of the proposed CRLA architecture. DRL agents are grouped into a cascaded structure of \(L\) levels. The state-action values of all agents are fed into a MixingNet implemented as a hyper-network (at bottom of the figure) to optimise the agents altogether. The paths of the gradient are shown as red arrows. Figure 2: An illustration of a tree-based structure for hierarchical action selection. The primitive action identifiers are located on the leaf nodes. Each internal node contains the action range of its children.