# Cascaded Reinforcement Learning Agents for Large Action Spaces in Autonomous Penetration Testing
## 4 Methodology
### Cooperative Multi-Agent Training  
The multi-agent training approach discussed in the previous section resembles the popular independent Q-learning (IQL) method [20] in which independent agents are trained simultaneously in the same environment. Despite its simplicity, IQL performs surprisingly well in practice and is usually a good benchmark against more advanced MARL algorithms [21]. The major drawback of IQL is that it does not consider non-stationarity in the policies of individual agents, nor do the agents take each other's policies or actions into account when updating their own policy. As a result, changes in observation are not the direct result of each agent's behaviour, and, hence, there is no convergence guarantee even with infinite exploration. To provide a sound theoretical guarantee of convergence with CRLA, we adapted a well-known multi-agent training network called QMIX [17]. QMIX trains an individual agent's network properly while taking the changing policies of the other agents into consideration. QMIX is shown in the right half of Figure 3 under the name Mixing Net. Generally, QMIX enables the training of a joint action value function by aggregating all the individual functions and optimising a combined action value function \(Q_{tot}(s,u)\) with \(u=f(a^{1},a^{2},\dots,a^{L})\). We had two reasons for choosing QMIX. First, we needed a coordination mechanism that would allow individual agents to account for changes to its own update as a result of another agent's policy. QMIX provides an end-to-end learning scheme that takes in all the values of individual action components (e.g., \(Q^{1}(s,a^{1}),\dots,Q^{L}(s,a^{L})\)), plus the state information, in order to learn an optimal \(Q^{*}_{tot}(s,u)\). It does this by minimising the following loss function:  
\[L(\theta)=\sum_{j=1}^{p}[(y_{j}^{tot}-Q_{tot}(s_{t},u;\theta)^{2}] \tag{7}\]  
and \(y^{tot}=r_{t}+\gamma\max_{u^{\prime}}Q(s^{\prime},u^{\prime};\theta^{-})\)  
Different functions and network architectures can then be used to construct \(Q_{tot}\)[22]. We tried to construct \(Q_{tot}\) through a summation function and a feed-forward neural network but, eventually, we selected a two-layered hyper-network to output the weight of the Mixing Net as this yielded a better result experimentally (Equation (8)):  
\[Q_{tot}(s_{t},u;\theta)=g(Q^{1}(s,a^{1}),\dots,Q^{L}(s,a^{L});\theta_{mixer}) \tag{8}\]  
This hyper-network was proposed by [PERSON] et al. [23] and used in [PERSON] et al. [17]. It has the advantages of using a small number of parameters \(W1\) and \(W2\) to output the weights for a larger network, in this case the Mixing Net. As a result, it helps to avoid a linear time complexity when scaling up to more agents or inputs. In previous studies on QMIX, the Mixing Net takes in a set of state-action values from homogenous agents as well as the environment's true states. This is different from our setting since each agent in the hierarchy can have a variable number of action components and we do not have access to the environment states. With respect to implementation, we had to concatenate \(h\) time steps of observations as the inputs to the Mixing Net. We also leveraged the vectorised computing feature in Pytorch [24] to efficiently adapt to the heterogenous input values. During back-propagation, the learning signal is distributed to each agent, which means they can consider other agents' policies when updating their own parameters to achieve a global optimum. This behaviour would not be possible without the use of this coordinating network. The second reason for using QMIX is that its parameters (\(W1\) and \(W2\) in Figure 3) are learned using a two-layer hyper-network with a non-negative activation function. This extra function is used to constrain the parameters of the Mixing Net to be positive, allowing QMIX to represent any factorisable and non-linear monotonic function \(\frac{\partial Q_{tot}}{\partial Q}>=0\) for all agents \(i\).