# Network Attack Path Selection and Evaluation Based on Q-Learning
## 2 FPN-Q Learning Algorithm to Determine the Best Attack Path
### Learning Stage  
In the traditional Q-Learning algorithm, the agent selects an action to act on the unknown environment during each iteration. After the environment receives the action, it generates an enhanced signal (reward or punishment) to feed back to the node. The node chooses the next action based on the enhanced signal and the new state of the environment. The principle of action selection is to increase the probability of receiving a positive reward. After continuous learning and trial and error, the node finds the optimal action control strategy and obtains cumulative returns. It is worth noting that the traditional Q-Learning algorithm randomly selects actions with equal probability in the exploration phase. However, in a network attack, the attacker has mastered all or part of the information about the security vulnerabilities of the information system before the attack. Therefore, the attack path will be selected based on experience rather than randomly selected with equal probability. The probability of each vulnerability being selected is related to the attack complexity: the higher the attack complexity, the smaller the chance of being selected. Therefore, this paper introduces the transition confidence of the FPN model \(\mu\) to optimize the exploration process of Q-Learning. In the exploration phase, the probability that the vulnerability \(j\) of host \(i\) is exploited \(p_{ij}\) is:  
\[p_{ij}=\frac{\frac{1}{\mu_{ij}}}{\sum\limits_{j=1}^{n}\frac{1}{\mu_{ij}}} \tag{1}\]  
where, \(n\) represents the number of exploitable vulnerabilities of host \(i\), \(\mu_{ij}\) is the attack complexity of exploitable vulnerability \(j\) of host \(i\), and its value range is (1,10). In the attack model, the attack cost is related to the attack complexity of the security vulnerability. Generally, the more complex the attack, the higher the attack cost. Therefore, this paper defines the attack complexity of security vulnerabilities as the attack cost. The attack proceeds are the threats to information systems caused by network attacks, which are related to the nature of the vulnerability itself. According to the FPN attack model established in Section 2.1, the following definitions are given:  
**Definition 1**. : _Initial single-step attack gain._The initial single-step attack gain is the threat to the system caused by the attacker invading host \(j\) through host \(i\) before the start of the learning process, expressed by the reward function \(g_{ij}\):  
\[g_{ij} =r_{ij}/\mu_{ij} \tag{2}\] \[r_{ij} =\alpha_{i}\cdot\alpha_{j} \tag{3}\]  
where, \(r_{ij}\) is the single-step attack reward, \(\alpha_{i}\) is the threat value caused by the intrusion system host \(i\) to the network, and \(\alpha_{j}\) is the threat value caused by the intrusion system host \(j\) to the network. **Definition 2**. : _Single-step cumulative attack gain._  
The cumulative single-step attack gain is the attack gain obtained by the attacker from host \(i\) invading host \(j\) after multiple intrusion learning, denoted by \(Q(h_{i},t_{ij},h_{j})\). \[Q(h_{i},t_{ij},h_{j})\leftarrow(1-\alpha)Q(h_{i},t_{ij},h_{j})+\alpha[g_{ij}+ \gamma\underset{t_{jk}\in T_{j}}{\max}Q(h_{j},t_{jk},h_{k})] \tag{4}\]  
where, \(\beta\) represents the learning factor, \(\gamma\) represents the discount factor between delayed return and immediate return, \(T_{j}\) represents the optional vulnerability of the next attack after the intrusion of host \(j\), and \(\underset{t_{jk}\in T_{j}}{\max}Q(h_{j},t_{jk},h_{k})\) represents the maximum gain that the attacker can obtain in the next attack after the host invades host \(j\). Based on the above definition, the attacker is regarded as the agent of the Q-Learning algorithm, and the information system is regarded as the environment where the attacker's attack behavior is given feedback.