# Defense Strategy Selection Model Based on Multistage Evolutionary Game Theory

Authors: Yanhua Liu; Hui Chen; Hao Zhang; Ximeng Liu
Journal: Security and Communication Networks
 (2021)

---

## Abstract

Evolutionary game theory is widely applied in network attack and defense. The existing network attack and defense analysis methods based on evolutionary games adopt the bounded rationality hypothesis. However, the existing research ignores that both sides of the game get more information about each other with the deepening of the network attack and defense game, which may cause the attacker to crack a certain type of defense strategy, resulting in an invalid defense strategy. The failure of the defense strategy reduces the accuracy and guidance value of existing methods. To solve the above problem, we propose a reward value learning mechanism (RLM). By analyzing previous game information, RLM automatically incentives or punishes the attack and defense reward values for the next stage, which reduces the probability of defense strategy failure. RLM is introduced into the dynamic network attack and defense process under incomplete information, and a multistage evolutionary game model with a learning mechanism is constructed. Based on the above model, we design the optimal defense strategy selection algorithm. Experimental results demonstrate that the evolutionary game model with RLM has better results in the value of reward and defense success rate than the evolutionary game model without RLM.


---

## Body

## 1. Introduction

The rapid development of IT infrastructures, such as cyber-physical systems and Internet of Things, has brought convenience to individuals and enterprises. But it also brings unprecedented security problems [1, 2]. Data management and communication layers in cyber-physical systems and the Internet of Things are vulnerable to cyberattacks such as DDoS attacks, APT, and vulnerability attacks, which seriously threaten network security [3]. According to the Crystal Market Research (CMR) report, to resist the increasingly severe attacks, the investment in the network security market is expected to increase from $ 58.13 billion to $ 173.57 billion from 2012 to 2022 [4]. Thus, it can be seen that the network attack and defense are increasingly severe, and network security defense has become an important problem to be solved in the field of network information [5]. Unfortunately, much research [6, 7] shows that improving information security technology alone cannot provide enough protection against persistent attacks. A new method is needed to guide the implementation of the defense strategy.Network attack and defense have the characteristics of opposite objectives and noncooperation relationships, which are consistent with the characteristics of the game. The game theory [8] applied to the network attack and defense process, from the perspective of the defender exploring how to get the best defense strategy, has become an important research direction of network security defense [9–12]. Traditional game theory assumes that both sides of the game are in a complete information game scene and are required to be rational completely [13, 14]. Complete information requires the game players to know the information of the entire environment [15]. Complete rationality assumes that game players can choose their own best game strategy after obtaining the other’s strategy and its revenue [16]. Evolutionary game theory [17, 18] starts from the condition of opaque information of players, takes the learning mechanism as the core, and influences the selection behavior of players through factors such as previous experience, learning, and imitating the behavior of others. Evolutionary game theory can better express the process of the mutual game between the attackers and defenders. It is widely used in the research work of the network attack and defense game [19–21].However, there are still some problems and challenges in applying evolutionary games in network attack and defense. (1) Existing studies using evolutionary game theory have introduced some relevant parameters to express evolutionary game ideas under incomplete information and bounded rationality [22, 23]. The use of these parameters is feasible in certain application scenes and shows a certain application value. However, the introduced parameters are calculated manually and need to be quantified by experts. There is currently no automatic calculation method. (2) In the multistage network attack and defense game, network attacks develop over time using new methods and targets [24], and the defense strategy may partially or completely fail. The existing evolutionary game model cannot effectively feed the failure information back to the next game stage, which leads to the shortcomings of the best defense strategy selection algorithm in terms of timeliness, accuracy, and efficiency. (3) Network attack and defense are a dynamic, multistage process [25]. The confrontation between attack and defense is not limited to one round or a certain stage.To address the above problems, we propose a reward value learning mechanism (RLM), a novel method for updating the reward value based on the game information in the previous stage. Inspired by machine learning, we use RLM to learn the attacker’s strategy and the change of its reward to predict the return value in the next stage. Furthermore, RLM is introduced to the evolutionary game model to select the best defense strategy for each stage. The main research work and contributions are as follows:(1)
Aiming at the multistage network attack and defense game scene, a reward value learning mechanism is designed. By updating the reward value of each stage, the mechanism improves the active defense ability when the defense strategy fails.(2)
Construct a new multistage evolutionary game model with a reward value learning mechanism, and solve the Nash equilibrium of each stage of the attack and defense game by constructing the replication dynamic equations (RD) of each stage.(3)
Based on the proposed multistage evolutionary game model, an optimal defense strategy selection algorithm is proposed. Experiments show that the algorithm can effectively improve the effective strategy selection probability, defense revenue, and defense success rate in a single defense strategy failure scene and multiple defense strategy failure scene.The remainder of this paper is organized as follows. Section2 briefly reviews evolutionary game theory and the related literature in network attack and defense. Section 3 describes the evolutionary game model based on Q-learning replication dynamic equations (QRD). Section 4 designs RLM and gives a new evolutionary game model based on RLM-QRD. Section 5 proposes the optimal defense strategy selection algorithm. Section 6 presents the experimental results for evaluating our model and compares it to the model without RLM. Finally, Section 7 concludes this paper and discusses the future works.


## 2. Related Works

The existing network defense works based on evolutionary games mainly include network defense based on the static evolutionary game and network defense based on the dynamic evolutionary game. The following are two aspects.


### 2.1. Static Network Defense Evolutionary Game

The static game assumes that the information of both sides of the game remains unchanged, and it is a one-shot game [26].Ruan et al. [27] established the attack and defense evolutionary game model using the lightweight broadcast authentication protocol to achieve security assurance and minimum resource cost. Abdalzaher et al. [28] used the scalability and low complexity of wireless sensor networks to propose a trust model that uses evolutionary game theory to make decisions to resist network attacks. Bouhaddi et al. [29] established a Bayesian game model to analyze the interaction between defenders and potential malicious nodes in the network. Aimed at the problems of free service users and users breaking system rules in peer-to-peer networks, Shareh et al. [30] established an evolutionary game model to resist network attacks from these two types of users. To explore and calculate more revenue of existing defense strategies, Jin et al. [31] combined Q-learning with replication dynamics equation to obtain Q-learning replication dynamics equation and proposed an evolutionary game model based on QRD. Aiming at the problem of the limited learning ability of players in the static network attack and defense, Liu et al. [32] established a network attack-defense evolutionary game model and designed an optimal defense strategy selection algorithm. Shi et al. [33] proposed an evolutionary game model based on honeypot technology to improve the security of the honeypot system.


### 2.2. Dynamic Network Defense Evolutionary Game

Compared with the static network attack and defense evolutionary game, the dynamic network attack and defense evolutionary game divides the game into multistage of attack and defense confrontation between the players, which is more in line with the network attack and defense.To realize the optimal defense decision in network attack and defense, Huang et al. proposed two dynamic evolutionary game models. One of the game models is the Markov-based time game model. This model selects the best defense strategy by constructing a revenue discount factor and all possible network system states [34]. Another game model used the best-response dynamic learning mechanism to study the evolutionary law of network defense strategy selection [35].To resist the invasion of the virus code, Hayel and Zhu [36] established an evolutionary Poisson game model by defining the number of players participating in the interaction to follow a Poisson process at a specific rate. Aiming at the security issues in radio networks, Fang et al. [37] established an evolutionary game model, using evolutionary stable strategy algorithms to defend dynamically against internal attacks. Mengibaev et al. [22] introduced parameters used to measure the dependence of game players on opponents into the evolutionary game model and applied them to the privacy protection of network users. Wang et al. [38] designed three evolutionary game models for different attack scenes in the dynamic network and introduced parameters that denote the degree of sensitivity of the players to the difference in revenue in these models.Hu et al. proposed different dynamic evolutionary game models for the problems in network attack and defense. In [39], a multistage Bayesian attack and defense evolutionary game model is proposed for the difficulty of selecting the optimal defense strategy in a dynamic confrontation network. At the same time, the selection intensity factor was introduced to improve the replication dynamic equation and enhance the randomness of the evolution process. More recently, to improve the timeliness and predictability of network attack and defense game, Hu et al. [40] proposed a dynamic evolutionary game model based on Logit Quantal Response Dynamics (LQRD), which introduced parameters into the evolutionary game to describe the rationality of attack and defense sides.


## 2.1. Static Network Defense Evolutionary Game

The static game assumes that the information of both sides of the game remains unchanged, and it is a one-shot game [26].Ruan et al. [27] established the attack and defense evolutionary game model using the lightweight broadcast authentication protocol to achieve security assurance and minimum resource cost. Abdalzaher et al. [28] used the scalability and low complexity of wireless sensor networks to propose a trust model that uses evolutionary game theory to make decisions to resist network attacks. Bouhaddi et al. [29] established a Bayesian game model to analyze the interaction between defenders and potential malicious nodes in the network. Aimed at the problems of free service users and users breaking system rules in peer-to-peer networks, Shareh et al. [30] established an evolutionary game model to resist network attacks from these two types of users. To explore and calculate more revenue of existing defense strategies, Jin et al. [31] combined Q-learning with replication dynamics equation to obtain Q-learning replication dynamics equation and proposed an evolutionary game model based on QRD. Aiming at the problem of the limited learning ability of players in the static network attack and defense, Liu et al. [32] established a network attack-defense evolutionary game model and designed an optimal defense strategy selection algorithm. Shi et al. [33] proposed an evolutionary game model based on honeypot technology to improve the security of the honeypot system.


## 2.2. Dynamic Network Defense Evolutionary Game

Compared with the static network attack and defense evolutionary game, the dynamic network attack and defense evolutionary game divides the game into multistage of attack and defense confrontation between the players, which is more in line with the network attack and defense.To realize the optimal defense decision in network attack and defense, Huang et al. proposed two dynamic evolutionary game models. One of the game models is the Markov-based time game model. This model selects the best defense strategy by constructing a revenue discount factor and all possible network system states [34]. Another game model used the best-response dynamic learning mechanism to study the evolutionary law of network defense strategy selection [35].To resist the invasion of the virus code, Hayel and Zhu [36] established an evolutionary Poisson game model by defining the number of players participating in the interaction to follow a Poisson process at a specific rate. Aiming at the security issues in radio networks, Fang et al. [37] established an evolutionary game model, using evolutionary stable strategy algorithms to defend dynamically against internal attacks. Mengibaev et al. [22] introduced parameters used to measure the dependence of game players on opponents into the evolutionary game model and applied them to the privacy protection of network users. Wang et al. [38] designed three evolutionary game models for different attack scenes in the dynamic network and introduced parameters that denote the degree of sensitivity of the players to the difference in revenue in these models.Hu et al. proposed different dynamic evolutionary game models for the problems in network attack and defense. In [39], a multistage Bayesian attack and defense evolutionary game model is proposed for the difficulty of selecting the optimal defense strategy in a dynamic confrontation network. At the same time, the selection intensity factor was introduced to improve the replication dynamic equation and enhance the randomness of the evolution process. More recently, to improve the timeliness and predictability of network attack and defense game, Hu et al. [40] proposed a dynamic evolutionary game model based on Logit Quantal Response Dynamics (LQRD), which introduced parameters into the evolutionary game to describe the rationality of attack and defense sides.


## 3. Preliminary

This section first introducesQ-learning and then replication dynamic equations. Finally, the definition of the evolutionary game model is proposed.


### 3.1. Q-Learning

Q-learning [41] is a reinforcement learning method, which can be regarded as an asynchronous dynamic programming method. Q-learning is also an adaptive value iteration method, which is based on the state-action value Qts′,a′ and guides the estimation of the state-action value Qt+1s,a at time t+1. Among them, the state-action value Qts,a is the expected revenue after action a is taken by state s at time t. State s′ is the state the learner reaches after using action a in state s. The Q-learning formula is given as follows:(1)Qt+1s,a⟵1−∂Qts,a+∂r+γmaxa′Qts′,a′.∂ is the usual step size parameter, r is immediate reinforcement, and γ is the discount factor.The principle ofQ-learning is to move in a discrete, finite state and select one from a finite set of actions every time, forming a controlled Markov process. Continuously, it aims to improve its evaluation of the quality of specific actions in a specific state to find a strategy with the most profit, which is consistent with the game’s goal.


### 3.2. Replication Dynamic Equation

The replication dynamic equation is a dynamic differential equation. It describes the frequency or probability of a certain strategy used in a specific group of people [42] and the degree to which the probability of the game’s main body choosing a strategy during the game. Its basic principle is that the game players gradually adopt more strategies with a revenue better than the average revenue. In addition, the replication dynamic equation can ensure that the evolutionary stable strategy is the Nash equilibrium, thereby obtaining the strategy that benefits the most. The replication dynamic equations of attackers and defenders in network attack and defense are given as follows:(2)xi′t=dxidt=xiQASi−Q¯AS.(3)yj′t=dyjdt=yjQDSj−Q¯DS.In the previous formulas,xi′t represents the change rate of the probability of the attacker selecting the attack strategy ASi over time, yj′t represents the change rate of the probability of the defender selecting the defense strategy DSj over time, QASi represents the expected revenue of the attacker’s selection of the attack strategy ASi, QDSj represents the expected revenue of the defender’s selection of the defense strategy DSj, Q¯AS represents the average revenue of the attack strategy set, and Q¯DS represents the average revenue of the defense strategy set.


### 3.3. Definition of Evolutionary Game Model Based on QRD

To extend the evolutionary game model to the dynamic network environment, this section introduces the stage definition into the evolutionary game model based on QRD. The model is defined below.Definition 1.
The evolutionary game model based on QRD is represented as 6 tuples, and its elements are defined as follows:(1)
N=NA,ND is the space of both sides in the game, and NA, ND are the attacker and the defender.(2)
K is the number of stages in the multistage attack and defense game; K=1,2,3,…,T.(3)
S=AS,DS is the strategy space of both sides in the game, where AS=AS1,AS2,…,ASn is the strategy set of the attacker, ASK is the strategy set of the attacker in stage K, DS=DS1,DS2,…,DSm is the strategy set of the defender, and DSK is the strategy set of the defender in stage K. n and m represent the number of strategies of the attacker and the defender, respectively. Therefore, both n and m are integers and n≥2,m≥2.(4)
θ=PA,PD is the belief set of attack and defense game, PA=x1,x2,…,xn is the probability distribution of the attacker’s overall strategy set AS; that is, any xi∈PA is the attacker’s choice of strategy ASi with probability xi to implement network attack, and PD=y1,y2,…,ym is the probability distribution of the defender’s overall strategy set DS; that is, any yj∈PD is the defender’s choice of strategy DSj with probability yi to implement defense strategy. In addition, the parameters satisfy the relationships 1≤i≤n, 1≤i≤n, ∑i=1nxi=1, ∑jmyj=1.(5)
Q=QA,QD is the revenue function set of the attack-defense game, and QA and QD represent the revenue function of the attacker and the defender, respectively, that is, the revenue of the attacker and the defender from the game strategy combination ASi,DSj.(6)
τ is the exploration factor of both sides of the game, indicating the degree of their exploration of game information. The larger the τ is, the greater the degree of exploration is and the more the attack and defense sides explore the unknown game information and make better decisions. The smaller the τ is, the smaller the degree of exploration is. The attack and defense sides mainly make the best decision based on the current known game information.


## 3.1. Q-Learning

Q-learning [41] is a reinforcement learning method, which can be regarded as an asynchronous dynamic programming method. Q-learning is also an adaptive value iteration method, which is based on the state-action value Qts′,a′ and guides the estimation of the state-action value Qt+1s,a at time t+1. Among them, the state-action value Qts,a is the expected revenue after action a is taken by state s at time t. State s′ is the state the learner reaches after using action a in state s. The Q-learning formula is given as follows:(1)Qt+1s,a⟵1−∂Qts,a+∂r+γmaxa′Qts′,a′.∂ is the usual step size parameter, r is immediate reinforcement, and γ is the discount factor.The principle ofQ-learning is to move in a discrete, finite state and select one from a finite set of actions every time, forming a controlled Markov process. Continuously, it aims to improve its evaluation of the quality of specific actions in a specific state to find a strategy with the most profit, which is consistent with the game’s goal.


## 3.2. Replication Dynamic Equation

The replication dynamic equation is a dynamic differential equation. It describes the frequency or probability of a certain strategy used in a specific group of people [42] and the degree to which the probability of the game’s main body choosing a strategy during the game. Its basic principle is that the game players gradually adopt more strategies with a revenue better than the average revenue. In addition, the replication dynamic equation can ensure that the evolutionary stable strategy is the Nash equilibrium, thereby obtaining the strategy that benefits the most. The replication dynamic equations of attackers and defenders in network attack and defense are given as follows:(2)xi′t=dxidt=xiQASi−Q¯AS.(3)yj′t=dyjdt=yjQDSj−Q¯DS.In the previous formulas,xi′t represents the change rate of the probability of the attacker selecting the attack strategy ASi over time, yj′t represents the change rate of the probability of the defender selecting the defense strategy DSj over time, QASi represents the expected revenue of the attacker’s selection of the attack strategy ASi, QDSj represents the expected revenue of the defender’s selection of the defense strategy DSj, Q¯AS represents the average revenue of the attack strategy set, and Q¯DS represents the average revenue of the defense strategy set.


## 3.3. Definition of Evolutionary Game Model Based on QRD

To extend the evolutionary game model to the dynamic network environment, this section introduces the stage definition into the evolutionary game model based on QRD. The model is defined below.Definition 1.
The evolutionary game model based on QRD is represented as 6 tuples, and its elements are defined as follows:(1)
N=NA,ND is the space of both sides in the game, and NA, ND are the attacker and the defender.(2)
K is the number of stages in the multistage attack and defense game; K=1,2,3,…,T.(3)
S=AS,DS is the strategy space of both sides in the game, where AS=AS1,AS2,…,ASn is the strategy set of the attacker, ASK is the strategy set of the attacker in stage K, DS=DS1,DS2,…,DSm is the strategy set of the defender, and DSK is the strategy set of the defender in stage K. n and m represent the number of strategies of the attacker and the defender, respectively. Therefore, both n and m are integers and n≥2,m≥2.(4)
θ=PA,PD is the belief set of attack and defense game, PA=x1,x2,…,xn is the probability distribution of the attacker’s overall strategy set AS; that is, any xi∈PA is the attacker’s choice of strategy ASi with probability xi to implement network attack, and PD=y1,y2,…,ym is the probability distribution of the defender’s overall strategy set DS; that is, any yj∈PD is the defender’s choice of strategy DSj with probability yi to implement defense strategy. In addition, the parameters satisfy the relationships 1≤i≤n, 1≤i≤n, ∑i=1nxi=1, ∑jmyj=1.(5)
Q=QA,QD is the revenue function set of the attack-defense game, and QA and QD represent the revenue function of the attacker and the defender, respectively, that is, the revenue of the attacker and the defender from the game strategy combination ASi,DSj.(6)
τ is the exploration factor of both sides of the game, indicating the degree of their exploration of game information. The larger the τ is, the greater the degree of exploration is and the more the attack and defense sides explore the unknown game information and make better decisions. The smaller the τ is, the smaller the degree of exploration is. The attack and defense sides mainly make the best decision based on the current known game information.


## 4. Game Model Based on RLM-QRD

To solve the problem of invalidation of specific defense strategies in network attack and defense scene, we put forward RLM with incentive and punishment mechanisms. RLM uses parameterα to calculate the attack and defense reward value in the next stage.


### 4.1. Definition of Game Model

By combining RLM with the evolutionary game based on QRD [31], this paper designs an evolutionary game model based on RLM-QRD; that is, RG=N,K,S,θ,Q,τ,α. N, K, S, θ, Q, and τ have been defined, and the definition of α is given below.Definition 2.
α is the incentive and punishment factor of reward value, which means the reward value of the corresponding strategy combination should be stimulated or punished when RLM is triggered. The value of α affects the probability of multistage strategy selection.
In the first stage of the game, the incentive and punishment factorα formula of reward value is as follows:(4)α=12×RV.
In stageK of the game, the formula of incentive and punishment factor α is as follows:(5)α=ANSN×RV.
The parameters in formulas (4) and (5) are defined as follows:(1)
RV is the reward variable, which represents the largest variable of the reward value in a single stage. Its value is determined by the influence of the other player’s strategy on itself. In general, RV is equal to the minimum reward value of the strategy combination of the defender.(2)
SN is the maximum number of learning stages of RLM, which denotes the maximum number of the learning stages the defender can learn from the previous game. Therefore, the maximum value of SN is the maximum number of stages of game T.(3)
AN is the number of a specific attack strategy in the past SN stages. If the attacker implements strategy ASi in the previous stage, then AN is equal to the number of ASi in the previous SN stages. The formula is as follows:(6)AN=numASi.


### 4.2. Framework of Game Model

The proposed framework of the multistage evolutionary game model is based on RLM-QRD, as shown in Figure1. It can be seen from Figure 1 that our model mainly includes an evolutionary game based on QRD and RLM.Figure 1
Framework of evolutionary game model based on RLM-QRD.The evolutionary game model based on QRD contains payoff quantification and QRD. Payoff quantification uses the information of the initial stage of the game to calculate the revenue of the attack and defense strategy in the initial stage. By QRD using the strategy revenue at the current stage, the optimal defense strategy is obtained by calculation.RLM is responsible for connecting all stages of the game. According to the known game information, it automatically incentives or punishes the attack and defense revenue of the next stage. Based on the above methods, the model can give a better defense strategy in the next stage.


### 4.3. Payoff Quantification of Attack and Defense Strategy

Network attack and defense strategy and its cost-reward analysis are the basis of achieving the optimal network security defense, so reasonable attack and defense payoff quantification affect the selection of defense strategy directly, thus affecting the defense effect. Here, we give some related definitions.Attack payoff matrixAM comprises attack revenue value aij generated by the attacker under attack and defense strategy combination ASi,DSj. According to the definition of the game model, the formula is as follows:(7)aij=QAASi,DSj=AR−AC.AR and AC represent attack revenue and attack cost, respectively.The attack payoff matrix in stageK is as follows:(8)AMK=a11Ka12K⋯a1mKa21Ka22K⋯a2mK⋮⋮⋱⋮an1Kan2K⋯anmK.Defense payoff matrixDM comprises defense revenue value dij generated by the defender under attack and defense strategy combination ASi,DSj. According to the definition of the game model, the formula is as follows:(9)dij=QDASi,DSj=DR−DC.DR and DC represent defense revenue and defense cost, respectively.The defense payoff matrix in stageK is as follows:(10)DMK=d11Kd12K…d1mKd21Kd22K…d2mK⋮⋮⋱⋮dn1Kdn2K…dnmK.


### 4.4. Q-Learning Replication Dynamic Equation

According to the Nash equilibrium theorem [43], it can be seen that, in a game with limited players in a limited strategy set, a mixed strategy Nash equilibrium must exist, and strategy x∗,y∗ is called a mixed strategy Nash equilibrium. When the game reaches Nash equilibrium, no player is worth changing his strategy unilaterally. In this case, if the attacker chooses strategy x∗ and the defender chooses strategy y∗, the attack and defense benefits are expressed as QAx∗,y∗ and QDx∗,y∗, respectively, satisfying the following conditions:(11)QAx∗,y∗≥QAx,y∗,∀x∈PA,QDx∗,y∗≥QDx∗,y,∀y∈PD.The following is the calculation of the expected revenueQASi of attack strategy ASi, the expected revenue QDSj of defense strategy DSj, the average revenue of attack strategy set Q¯AS, and the average revenue of defense strategy set Q¯DS:(12)QASi=∑j=1maijyj,QDSj=∑i=1ndijxi,Q¯AS=∑i=1n∑j=1maijxiyj,Q¯DS=∑j=1m∑i=1ndijxiyj.The Boltzmann probability distribution is used to represent the attack and defense strategy, and theQ-learning algorithm is introduced into the replication dynamic equation to get the QRD equation. The probability of strategy selection for QRD is given as follows:(13)xik=expτ⋅QASik∑l=1nexpτ⋅QASik.(14)yjk=expτ⋅QDSjk∑l=1mexpτ⋅QDSlk.Here,xik and yjk obey the Boltzmann probability distribution. xik denotes the probability that the attacker selects the attack strategy ASi in the k-th attack and defense confrontation at the same game stage. yjk denotes the probability that the defender selects the defense strategy DSj in the k-th attack and defense confrontation at the same game stage. QASik denotes the expected revenue obtained by the attacker choosing the attack strategy ASi in the k-th attack and defense confrontation at the same game stage. QDSjk denotes the expected revenue obtained by the defender choosing the defense strategy DSj in the k-th attack and defense confrontation at the same stage. The Q-learning replicated dynamic equations formulas (15) and (16) are derived from the correlation formulas (2), (3), (15), and (14):(15)x′t=dxidt=xiQASi−Q¯AS︸RD+1τxi∑k=1nxklnxk/xi︸ME.(16)y′t=dyjdt=yjQDSj−Q¯DS︸RD+1τyj∑l=1myllnyl/yj︸ME.QRD consists of replication dynamic equation (RD) and mutation equation (ME). RD selects the most profitable strategy under current information. ME is to try different new strategies in unknown network attack and defense scenes, and constantly try and make error, and learn and adjust the strategies, which better reflects the diversity and uncertainty of network attack and defense.From the definition of evolutionary equilibrium, when the strategies of players reach evolutionary equilibrium, there is(17)x′t=0andy′t=0.Solutionx∗,y∗ of the above formula is an evolutionary stable equilibrium point. At this time, the τ value in the above equation needs to be large enough to make the selection probability of each strategy stable.


### 4.5. Reward Value Learning Mechanism

As shown in Algorithm1, RLM calculates the incentive and punishment factor α according to reward variable RV and the proportion of the number AN of a certain type of attack strategy in the past SN stage. According to α and the defense result R of the last stage, RLM changes the reward value of the corresponding attack and defense strategy to change the attack and defense reward value of the next stage.Algorithm 1: Reward value learning mechanism.
Input: revenue of strategy combination in the last stage AMK01,DMK01. Current stage K attack and defense strategy in the last stage ASiK01,DSjK01. Defense results of the last stage R.Output: revenue value of each strategy combination in the current stage AMK,DMK.(1)
InitializeSN, AN, RV(2)
ifK=1then(3)
Calculateα from equation (4)(4)
else ifK>1then(5)
Calculateα from equation (5)(6)
end if(7)
ifR=0//The result of the last stage of defense was failure then(8)
aijK=aijK−1+α and dijK=dijK−1−α(9)
else(10)
aijK=aijK−1−α and dijK=dijK−1+α(11)
end if(12)
returnAMK,DMKIf the defense successfully resists an attack in the last stageR=1, RLM increases the defense reward value of the specific strategy combination dijK, and decreases the attack reward value of the specific strategy combination aijK. If the defense failed in the last stage R=0, RLM decreases the defense reward value dijK of the specific strategy combination and increases the attack reward value aijK of the specific strategy combination in the last stage.


## 4.1. Definition of Game Model

By combining RLM with the evolutionary game based on QRD [31], this paper designs an evolutionary game model based on RLM-QRD; that is, RG=N,K,S,θ,Q,τ,α. N, K, S, θ, Q, and τ have been defined, and the definition of α is given below.Definition 2.
α is the incentive and punishment factor of reward value, which means the reward value of the corresponding strategy combination should be stimulated or punished when RLM is triggered. The value of α affects the probability of multistage strategy selection.
In the first stage of the game, the incentive and punishment factorα formula of reward value is as follows:(4)α=12×RV.
In stageK of the game, the formula of incentive and punishment factor α is as follows:(5)α=ANSN×RV.
The parameters in formulas (4) and (5) are defined as follows:(1)
RV is the reward variable, which represents the largest variable of the reward value in a single stage. Its value is determined by the influence of the other player’s strategy on itself. In general, RV is equal to the minimum reward value of the strategy combination of the defender.(2)
SN is the maximum number of learning stages of RLM, which denotes the maximum number of the learning stages the defender can learn from the previous game. Therefore, the maximum value of SN is the maximum number of stages of game T.(3)
AN is the number of a specific attack strategy in the past SN stages. If the attacker implements strategy ASi in the previous stage, then AN is equal to the number of ASi in the previous SN stages. The formula is as follows:(6)AN=numASi.


## 4.2. Framework of Game Model

The proposed framework of the multistage evolutionary game model is based on RLM-QRD, as shown in Figure1. It can be seen from Figure 1 that our model mainly includes an evolutionary game based on QRD and RLM.Figure 1
Framework of evolutionary game model based on RLM-QRD.The evolutionary game model based on QRD contains payoff quantification and QRD. Payoff quantification uses the information of the initial stage of the game to calculate the revenue of the attack and defense strategy in the initial stage. By QRD using the strategy revenue at the current stage, the optimal defense strategy is obtained by calculation.RLM is responsible for connecting all stages of the game. According to the known game information, it automatically incentives or punishes the attack and defense revenue of the next stage. Based on the above methods, the model can give a better defense strategy in the next stage.


## 4.3. Payoff Quantification of Attack and Defense Strategy

Network attack and defense strategy and its cost-reward analysis are the basis of achieving the optimal network security defense, so reasonable attack and defense payoff quantification affect the selection of defense strategy directly, thus affecting the defense effect. Here, we give some related definitions.Attack payoff matrixAM comprises attack revenue value aij generated by the attacker under attack and defense strategy combination ASi,DSj. According to the definition of the game model, the formula is as follows:(7)aij=QAASi,DSj=AR−AC.AR and AC represent attack revenue and attack cost, respectively.The attack payoff matrix in stageK is as follows:(8)AMK=a11Ka12K⋯a1mKa21Ka22K⋯a2mK⋮⋮⋱⋮an1Kan2K⋯anmK.Defense payoff matrixDM comprises defense revenue value dij generated by the defender under attack and defense strategy combination ASi,DSj. According to the definition of the game model, the formula is as follows:(9)dij=QDASi,DSj=DR−DC.DR and DC represent defense revenue and defense cost, respectively.The defense payoff matrix in stageK is as follows:(10)DMK=d11Kd12K…d1mKd21Kd22K…d2mK⋮⋮⋱⋮dn1Kdn2K…dnmK.


## 4.4. Q-Learning Replication Dynamic Equation

According to the Nash equilibrium theorem [43], it can be seen that, in a game with limited players in a limited strategy set, a mixed strategy Nash equilibrium must exist, and strategy x∗,y∗ is called a mixed strategy Nash equilibrium. When the game reaches Nash equilibrium, no player is worth changing his strategy unilaterally. In this case, if the attacker chooses strategy x∗ and the defender chooses strategy y∗, the attack and defense benefits are expressed as QAx∗,y∗ and QDx∗,y∗, respectively, satisfying the following conditions:(11)QAx∗,y∗≥QAx,y∗,∀x∈PA,QDx∗,y∗≥QDx∗,y,∀y∈PD.The following is the calculation of the expected revenueQASi of attack strategy ASi, the expected revenue QDSj of defense strategy DSj, the average revenue of attack strategy set Q¯AS, and the average revenue of defense strategy set Q¯DS:(12)QASi=∑j=1maijyj,QDSj=∑i=1ndijxi,Q¯AS=∑i=1n∑j=1maijxiyj,Q¯DS=∑j=1m∑i=1ndijxiyj.The Boltzmann probability distribution is used to represent the attack and defense strategy, and theQ-learning algorithm is introduced into the replication dynamic equation to get the QRD equation. The probability of strategy selection for QRD is given as follows:(13)xik=expτ⋅QASik∑l=1nexpτ⋅QASik.(14)yjk=expτ⋅QDSjk∑l=1mexpτ⋅QDSlk.Here,xik and yjk obey the Boltzmann probability distribution. xik denotes the probability that the attacker selects the attack strategy ASi in the k-th attack and defense confrontation at the same game stage. yjk denotes the probability that the defender selects the defense strategy DSj in the k-th attack and defense confrontation at the same game stage. QASik denotes the expected revenue obtained by the attacker choosing the attack strategy ASi in the k-th attack and defense confrontation at the same game stage. QDSjk denotes the expected revenue obtained by the defender choosing the defense strategy DSj in the k-th attack and defense confrontation at the same stage. The Q-learning replicated dynamic equations formulas (15) and (16) are derived from the correlation formulas (2), (3), (15), and (14):(15)x′t=dxidt=xiQASi−Q¯AS︸RD+1τxi∑k=1nxklnxk/xi︸ME.(16)y′t=dyjdt=yjQDSj−Q¯DS︸RD+1τyj∑l=1myllnyl/yj︸ME.QRD consists of replication dynamic equation (RD) and mutation equation (ME). RD selects the most profitable strategy under current information. ME is to try different new strategies in unknown network attack and defense scenes, and constantly try and make error, and learn and adjust the strategies, which better reflects the diversity and uncertainty of network attack and defense.From the definition of evolutionary equilibrium, when the strategies of players reach evolutionary equilibrium, there is(17)x′t=0andy′t=0.Solutionx∗,y∗ of the above formula is an evolutionary stable equilibrium point. At this time, the τ value in the above equation needs to be large enough to make the selection probability of each strategy stable.


## 4.5. Reward Value Learning Mechanism

As shown in Algorithm1, RLM calculates the incentive and punishment factor α according to reward variable RV and the proportion of the number AN of a certain type of attack strategy in the past SN stage. According to α and the defense result R of the last stage, RLM changes the reward value of the corresponding attack and defense strategy to change the attack and defense reward value of the next stage.Algorithm 1: Reward value learning mechanism.
Input: revenue of strategy combination in the last stage AMK01,DMK01. Current stage K attack and defense strategy in the last stage ASiK01,DSjK01. Defense results of the last stage R.Output: revenue value of each strategy combination in the current stage AMK,DMK.(1)
InitializeSN, AN, RV(2)
ifK=1then(3)
Calculateα from equation (4)(4)
else ifK>1then(5)
Calculateα from equation (5)(6)
end if(7)
ifR=0//The result of the last stage of defense was failure then(8)
aijK=aijK−1+α and dijK=dijK−1−α(9)
else(10)
aijK=aijK−1−α and dijK=dijK−1+α(11)
end if(12)
returnAMK,DMKIf the defense successfully resists an attack in the last stageR=1, RLM increases the defense reward value of the specific strategy combination dijK, and decreases the attack reward value of the specific strategy combination aijK. If the defense failed in the last stage R=0, RLM decreases the defense reward value dijK of the specific strategy combination and increases the attack reward value aijK of the specific strategy combination in the last stage.


## 5. Optimal Defense Strategy Selection Algorithm

In this paper, the Nash equilibrium solution of the multistage evolutionary game is regarded as a set of equilibrium solutions of a multistage evolutionary game. Each stage learns the known game information through the reward value learning mechanism to change the reward value of defense strategy in the current stage. According to the optimal defense strategy of each stage, the multistage optimal defense strategy set is constructed.The multistage network attack and defense game tree based on our game model is shown in Figure2. The black dots in Figure 2 represent the attacker at stage K, picked out with probability PA=x1,x2,…,xn and executed the attack strategy AS=AS1,AS2,…,ASn. The blue dots denote the defender at stage K, picked out with probability PD=y1,y2,…,ym and executed the defensive strategy DS=DS1,DS2,…,DSm.Figure 2
Multistage network attack and defense game tree.As shown in Figure2, the process of multistage network attack and defense games is as follows:(1)
In the initial stage of a network attack and defense game, the corresponding model parameters and reward values are calculated based on the current information of players, and the respective optimal decision in the current stage is reached by solving the Nash equilibrium and conserving information such as game resultsR of the current stage, the strategy combination ASiK,DSjK, and the attack and defense revenue AMK,DMK for use in the calculation of the attack and defense revenue at the next stage.(2)
When the network attack and defense games enter stageKK>1, as K grows larger and the attacker and defender gain more information gradually, the players of the game tend to be completely rational, and the revenue of the various combinations of strategies is highly likely to change. At this time, according to Algorithm 1, the incentive and punishment factor α of this stage is calculated, the attack and defense revenue AMK,DMK of this stage is obtained from α and the information saved in the previous stage. The optimal defense strategy of the current stage is solved through the Nash equilibrium solution, and the defense result R, attack and defense strategy ASiK,DSjK , and attack and defense revenue AMK,DMK of this stage are kept.Next, we propose an optimal defense strategy selection algorithm based on the evolutionary game model based on RLM-QRD.Algorithm 2: The optimal defense strategy selection algorithm based on RLM-QRD evolutionary game.
Input: evolutionary game model based on RLM-QRD.Output: probability set of optimal defense strategy in K-th stage PDK.(1)
InitializeRG=N,K,S,θ,Q,τ,α(2)
fori⟵1tondo(3)
forj⟵1tomdoCalculateAMK,DMK from equations (8) and (10)(4)
end for(6)
forK⟵1toTdo(7)
forj⟵1tomdo(8)
Constructy′t from equation (16)(9)
end for(10)
fori⟵1tondo(11)
Constructx′t from equation (15)(12)
end for(13)
Calculateτ and PDK from equation (17)(14)
CalculateAMK+1,DMK+1 from Algorithm 1(15)
OutputPDK=y1K,y2K,…,yMK(16)
end forAs shown in Algorithm2, if K is the number of stages of the game, n and m represent the number of strategies of the attacker and the defender, respectively; generally, there is K>0,n>0,m>0. The time cost of Algorithm 2 in each stage is mainly concentrated on Step 8 and Step 11. In Step 8, the QRD of each defense strategy is constructed in turn, and the computational complexity of solving the equation in Step 8 is Om2+nm. In Step 11, the QRD of each attack strategy is constructed in turn, and the computational complexity of solving the equation in Step 11 is On2+nm. The total time complexity of the algorithm is OKm+n2. The storage cost of the algorithm mainly focuses on the storage of the payoff matrix. The storage of the payoff matrix has high complexity, which contains the total number of nm memory cells. Therefore, the spatial complexity is Onm.Table1 shows the comprehensive comparison between our model and other models in the literature. The following are some discussions:(1)
Payoff quantification: references [31, 36] have no extra parameters for the payoff quantification. References [22, 23] and [40] introduce specific parameters into the game model and set those parameters to quantify the attack and defense payoff in their respective application scenarios. We also introduce some parameters and calculate the parameters by the past game information, to better quantify the attack and defense payoff under the scenario of defense strategy failure.(2)
Equilibrium solution: the equilibrium solution represents the method used to solve the Nash equilibrium in the game model. References [23, 36] use RD, and reference [22] uses fermi function. All of these methods have had some success in their application. References [31, 40] improve on RD by proposing QRD and LQRD, respectively, and have been successful in optimal defense strategy selection. For the scenario of defense strategy failure, we propose RLM-QRD, which aims to realize automatic calculation of revenue, maximize defense revenue, and select the optimal defense strategy.(3)
Game type and Algorithm complexity: reference [31] is a static game, which has the advantage of low algorithm complexity. Although the complexity of the dynamic game algorithm is high, the dynamic game is more suitable for network attack and defense.Table 1
Comprehensive comparison among our model and other models.
Ref.Payoff quantificationGame typeEquilibrium solutionAlgorithm complexityApplication[31]No extra parameterStaticQRDOn+m2Defense strategy selection[36]No extra parameterDynamicRDOKm+n2Virus protection[23]Discount factorδDynamicRDOKm+n2Defense strategy selection[22]Dependent parameterμDynamicFermi functionOKm+n2Privacy protection[40]Rational parameterλDynamicLQRDOKm+n2Defense strategy selectionOursRLMDynamicRLM-QRDOKm+n2Defense strategy selection under strategy failureBased on the above discussion, our model is more suitable for failure scenarios of defense strategies in dynamic network attack and defense.


## 6. Experiment and Analysis

In this section, we verify the effectiveness of our model in the scenario of policy failure. Firstly, we give the attack strategy set and defense strategy set. We also introduce the strategy failure scenes. Secondly, we calculate the exploration factorτ. Thirdly, we analyze the defense strategy selection probability of our model under the single strategy failure scene and multistrategies failure scene. Fourthly, we compare the revenue of our model with those of the model without RLM. Finally, we compare the defense success rate of our model with those of the model without RLM.


### 6.1. Experimental Setup

In our experiment, the attack and defense behavior database of MIT [44] and China National Vulnerability Database of Information Security (CNNVD) [45] are used to analyze the attack and defense atomic strategy, as shown in Tables 2 and 3.Table 2
Attack strategy and sequence atom attack strategy.
NumberNameAttack strategyAS1AS2a1Install Web√a2Remote attack√a3Obtain user privileges√a4Buffer overflow attack√a5Homepage attack√Table 3
Defense strategy and sequence atom defense strategy.
NumberNameDefense strategyDS1DS2d1Limit packers from ports√d2Modify account password√d3Restart database server√d4Limit packets from ports√d5Reinstall listener program√√d6Correct homepage√The attacker makes use of the vulnerability in the network information system to choose some atomic attack strategies. The defender selects several atomic defense strategies to defend against network attacks [46]. The attack and defense strategies in this experiment are composed of several atomic attack and defense strategies. For both sides of network attack and defense, set attack strategies AS1=a1,a2,a5 and AS2=a3,a4 and defense strategies DS1=d1,d3,d5,d6 and DS2=d2,d4,d5. By referencing [31, 34] and the definitions of attack and defense reward and cost in Section 4.3, the attack and defense revenue matrix of the first stage is given, as shown in Tables 4 and 5. The larger the number in the table, the greater the revenue from attack or defense.Table 4
Attack revenue matrix of the first stage.
Attack strategyDefense strategyDS1DS2AS110070AS26080Table 5
Defense revenue matrix of the first stage.
Attack strategyDefense strategyDS1DS2AS17010AS25060In summary, we set the probability that attack strategyAS=AS1,AS2 and defense strategy DS=DS1,DS2 are chosen as PA=x,1−x and PD=y,1−y, respectively, and set the reward variable RV=10. Since there is a logarithm in the Q-learning replication dynamic equation, we assume that the value of the probability of strategy selection ranges 0.01,0.99. To better show the results of the experiment, the maximum number of learning stages SN=500 and the maximum number of stages T=1000 of RLM are set in this experiment.This experiment verifies the validity of the evolutionary game model based on RLM-QRD in the scene of specific defense strategy failure from the perspective of single defense strategy failure and multiple defense strategies failure.This experiment assumes that the attacker has acquired a specific defense strategy method at a certain stage and proposes and implements a new attack strategy and its selection probability, which results in the invalidation of the defense strategy. As shown in Table6, status I and status II are single defense strategy failure scenes, and status III is multiple defense strategies failure scene.Table 6
Strategy invalidation scene.
StatusDescriptionIThe attacker selects attack strategyAS1 to make defense strategy DS1 invalidIIThe attacker randomly selects attack strategyAS to make defense strategy DS1 invalidIIIThe attacker randomly selects attack strategyAS to make defense strategy DS invalid randomly


### 6.2. The Calculation of Exploration Factor

It can be seen from equations (5) and (6) that when the exploration factor τ is small, both sides of the game do not fully grasp each other’s relevant information under the condition of one stage. The ME in the Q-learning replication dynamic equation has a better impact, and the probability of attack strategies and defense strategies (AS1, AS2, DS1, DS2) selection is unstable.Figure3 shows the influence of exploration factor τ on the attack and defense strategy evolution. As shown in Figure 3, with the acquisition and analysis of each other’s information in a single stage, the exploration factor τ gradually increases. The effect of the mutation equation decreases, the replication dynamic equation gradually begins to play a bigger role. The probability of attack and defense strategy selection gradually tends to be stable. QRD gradually degenerates into the replication dynamic equation. The simulation results show that the selection probability of attack and defense strategy remains constant when τ≥3.Figure 3
The influence of exploration factorτ on attack and defense strategy evolution.To sum up, the larger the exploration factorτ is, the more information the offensive and defensive sides can finally obtain in this game stage. Usually, both sides of the game will change their strategies only when they know enough game information, and the game will enter a new stage. Therefore, this experiment sets the exploration factor τ=100.


### 6.3. Probability of Defense Strategy Selection Experiment

To verify that the model can reduce the selection probability of failure strategy and increase the selection probability of effective strategy in single strategy failure and multistrategies failure scenes. This section studies the selection probability of optimal defense strategy when the status is I, II, and III, and the results are shown in Figures4 and 5. Each point represents a stage. The red points and the orange points represent the failure of defense, which means the defender chooses the failure strategy. The green points and the blue points represent the success of the defense, which means the defender chooses the effective strategy. The x-axis and y-axis are the selection probabilities of game stage K and defense strategy DS1, respectively.Figure 4
Selection probability of defense strategy under status numbers I and II.Figure 5
Selection probability of defense strategy under status number III.


### 6.4. Single Strategy Failure Scene

Figure4 shows the selection probability of defense strategy DS1 in the number of stages K=30 when the status is I and II. As shown in Figure 4, the selection probability of defense strategy DS1 converges to around 0.01 after only about K=10 stages in status I because defense strategy DS1 has failed. The model implements defense strategy DS2 with a high probability to resist the new attack strategy. The selection probability of defense strategy DS1 converges after stage K=15 in status II. Compared with the selection probability of defense strategy in status I, the selection probability of defense strategy in status II converges slowly because any strategy of the attacker can make defense strategy DS1 invalid when the status is II.As can be seen from Figure4, the probability of failed defense strategy DS1 shows a short upward trend before convergence and defense failure occur in the early stage in status I and status II because RLM is still learning attack strategies and their benefits at the beginning of the game stage. Due to the change of attack strategy, the strategy given by the defender is not optimal at this time. After several short stages of learning, RLM will converge the probability of failure strategy to 0.01. The defender has obtained the optimal defense strategy through several stages of learning. The game defense is successful.


### 6.5. Multistrategies Failure Scene

In status III, the attacker can choose any attack strategy to make any defense strategy invalid. In this scene, the defense strategy that failed may beDS1 in stage K, and the defense strategy that failed may become DS2 in stage K+1. So, the game becomes more complex.Figure5 shows the defense strategy selection probability in status III. As shown in Figure 5, the change of defense strategy selection probability is unstable, which is caused by the high complexity of the network state in this scene. In this scene, the defense side tends to give a more selection probability to the defense strategy DS1.In status III, the failed strategies of each stage are not necessarily the same, so we cannot judge the model from the probability of defense strategy selection experiment. However, the model can be verified from attack and defense revenue experiment and defense success rate experiment in status III.


### 6.6. Attack and Defense Revenue Experiment

Figures6(a), 6(b), and 7 show the comparative diagram of attack and defense revenue values based on our model and the evolutionary game model without RLM in statuses I, II, and III. The x-axis and y-axis are game stage and attack and defend revenue. The blue line and the yellow line represent the defense and the attack revenue of our model. The green line and the red line represent the defense and the attack revenue of the evolutionary game model without RLM.Figure 6
Comparison of attack and defense revenue under status I and II. (a) Status I. (b) Status II.
(a)(b)Figure 7
Comparison of attack and defense revenue under status III.


### 6.7. Single Strategy Failure Scene

Figures6(a) and 6(b) show the comparison of the attack and defense revenue values of the game model in the number of stages K=30 when the status is I and II.It can be seen from Figures6(a) and 6(b) that the difference between the attack and defense revenues of the two models is not obvious at the beginning of the game stage, which indicates that RLM has not fully learned the attacker’s information. At this time, the attack and defense revenues of the two models are unstable. Even because of the change of attack strategy, the revenues of the attacker may become more, and the revenues of the defender may become less.After several stages, the defense revenues of the two models are positively correlated with the game stage, and the attack revenues are negatively correlated with the game stage. Compared with the game model without RLM, the model in Figure6(a) has higher defense revenue and lower attack revenue after stage K=10, and the model in Figure 6(b) has lower attack revenue after stage K=15 and higher defense revenue after stage K=20. Combined with the previous analysis, we can conclude that the revenue value in the game model with RLM changes drastically from the probability convergence of defense strategy. This situation also means that after the short-term learning stages of RLM, the defense revenue begins to rise, and the attack revenue begins to decline.Compared with Figure6(a), the change of attack and defense revenue in Figure 6(b) is relatively slow because any attacker’s strategy may make defense strategy DS1 invalid when the status is II. To sum up, this model can more effectively resist the attackers in the scenes of statuses I and II.


### 6.8. Multistrategies Failure Scene

Figure7 shows the comparison of attack and defense revenue between our game model and the game model without RLM when the status is III. From the previous description, we can conclude that the scene is highly complex, and the attacker can adjust the strategy at each stage to invalidate the specific defense strategy. Therefore, with the continuous change of attack strategy, the attack and defense revenues fluctuate. Next, from the perspective of attack and defense revenues, we verify that our model has better defense ability than the model without RLM in status III.As can be seen from Figure7, the defense revenue of our model and the attack revenue of the game model without RLM rise with fluctuation, and the attack revenue of our model and the defense revenue of the game model without RLM decline with fluctuation. From stage K=400, the defense revenue of our model is always higher than that of the game model without RLM. From stage K=300, the attack revenue of this model is always lower than that of the game model without RLM.To sum up, after learning multiple stages with RLM, the model can make the defense revenue greater and the attack revenue lower, to better resist network attacks.


### 6.9. Defense Success Rate Experiment

Figure8 shows the comparison of defense success rate between our model and the game model without RLM in stage K=1000. The green column denotes the number of successful defense stages when the models choose defense strategy DS1. The blue column denotes the number of successful defense stages when the models choose defense strategy DS2. The red column denotes the number of failed defense stages when the models choose defense strategy DS1. The yellow column denotes the number of failed defense stages when the models choose defense strategy DS2.Figure 8
Comparison of defense success rate.It can be seen from Figure8 that our model under statuses I, II, and III has a higher defense success rate than the game model without RLM. Compared with the game model without RLM, the defense success rate of this model is about 22.5% higher under state I. The defense success rate of our model increases by about 23.5% under status II. Our model improves by about 7.4% under status III. In conclusion, compared with the evolutionary game model without RLM, our model can better select the best defense strategy in the above three scenes.


## 6.1. Experimental Setup

In our experiment, the attack and defense behavior database of MIT [44] and China National Vulnerability Database of Information Security (CNNVD) [45] are used to analyze the attack and defense atomic strategy, as shown in Tables 2 and 3.Table 2
Attack strategy and sequence atom attack strategy.
NumberNameAttack strategyAS1AS2a1Install Web√a2Remote attack√a3Obtain user privileges√a4Buffer overflow attack√a5Homepage attack√Table 3
Defense strategy and sequence atom defense strategy.
NumberNameDefense strategyDS1DS2d1Limit packers from ports√d2Modify account password√d3Restart database server√d4Limit packets from ports√d5Reinstall listener program√√d6Correct homepage√The attacker makes use of the vulnerability in the network information system to choose some atomic attack strategies. The defender selects several atomic defense strategies to defend against network attacks [46]. The attack and defense strategies in this experiment are composed of several atomic attack and defense strategies. For both sides of network attack and defense, set attack strategies AS1=a1,a2,a5 and AS2=a3,a4 and defense strategies DS1=d1,d3,d5,d6 and DS2=d2,d4,d5. By referencing [31, 34] and the definitions of attack and defense reward and cost in Section 4.3, the attack and defense revenue matrix of the first stage is given, as shown in Tables 4 and 5. The larger the number in the table, the greater the revenue from attack or defense.Table 4
Attack revenue matrix of the first stage.
Attack strategyDefense strategyDS1DS2AS110070AS26080Table 5
Defense revenue matrix of the first stage.
Attack strategyDefense strategyDS1DS2AS17010AS25060In summary, we set the probability that attack strategyAS=AS1,AS2 and defense strategy DS=DS1,DS2 are chosen as PA=x,1−x and PD=y,1−y, respectively, and set the reward variable RV=10. Since there is a logarithm in the Q-learning replication dynamic equation, we assume that the value of the probability of strategy selection ranges 0.01,0.99. To better show the results of the experiment, the maximum number of learning stages SN=500 and the maximum number of stages T=1000 of RLM are set in this experiment.This experiment verifies the validity of the evolutionary game model based on RLM-QRD in the scene of specific defense strategy failure from the perspective of single defense strategy failure and multiple defense strategies failure.This experiment assumes that the attacker has acquired a specific defense strategy method at a certain stage and proposes and implements a new attack strategy and its selection probability, which results in the invalidation of the defense strategy. As shown in Table6, status I and status II are single defense strategy failure scenes, and status III is multiple defense strategies failure scene.Table 6
Strategy invalidation scene.
StatusDescriptionIThe attacker selects attack strategyAS1 to make defense strategy DS1 invalidIIThe attacker randomly selects attack strategyAS to make defense strategy DS1 invalidIIIThe attacker randomly selects attack strategyAS to make defense strategy DS invalid randomly


## 6.2. The Calculation of Exploration Factor

It can be seen from equations (5) and (6) that when the exploration factor τ is small, both sides of the game do not fully grasp each other’s relevant information under the condition of one stage. The ME in the Q-learning replication dynamic equation has a better impact, and the probability of attack strategies and defense strategies (AS1, AS2, DS1, DS2) selection is unstable.Figure3 shows the influence of exploration factor τ on the attack and defense strategy evolution. As shown in Figure 3, with the acquisition and analysis of each other’s information in a single stage, the exploration factor τ gradually increases. The effect of the mutation equation decreases, the replication dynamic equation gradually begins to play a bigger role. The probability of attack and defense strategy selection gradually tends to be stable. QRD gradually degenerates into the replication dynamic equation. The simulation results show that the selection probability of attack and defense strategy remains constant when τ≥3.Figure 3
The influence of exploration factorτ on attack and defense strategy evolution.To sum up, the larger the exploration factorτ is, the more information the offensive and defensive sides can finally obtain in this game stage. Usually, both sides of the game will change their strategies only when they know enough game information, and the game will enter a new stage. Therefore, this experiment sets the exploration factor τ=100.


## 6.3. Probability of Defense Strategy Selection Experiment

To verify that the model can reduce the selection probability of failure strategy and increase the selection probability of effective strategy in single strategy failure and multistrategies failure scenes. This section studies the selection probability of optimal defense strategy when the status is I, II, and III, and the results are shown in Figures4 and 5. Each point represents a stage. The red points and the orange points represent the failure of defense, which means the defender chooses the failure strategy. The green points and the blue points represent the success of the defense, which means the defender chooses the effective strategy. The x-axis and y-axis are the selection probabilities of game stage K and defense strategy DS1, respectively.Figure 4
Selection probability of defense strategy under status numbers I and II.Figure 5
Selection probability of defense strategy under status number III.


## 6.4. Single Strategy Failure Scene

Figure4 shows the selection probability of defense strategy DS1 in the number of stages K=30 when the status is I and II. As shown in Figure 4, the selection probability of defense strategy DS1 converges to around 0.01 after only about K=10 stages in status I because defense strategy DS1 has failed. The model implements defense strategy DS2 with a high probability to resist the new attack strategy. The selection probability of defense strategy DS1 converges after stage K=15 in status II. Compared with the selection probability of defense strategy in status I, the selection probability of defense strategy in status II converges slowly because any strategy of the attacker can make defense strategy DS1 invalid when the status is II.As can be seen from Figure4, the probability of failed defense strategy DS1 shows a short upward trend before convergence and defense failure occur in the early stage in status I and status II because RLM is still learning attack strategies and their benefits at the beginning of the game stage. Due to the change of attack strategy, the strategy given by the defender is not optimal at this time. After several short stages of learning, RLM will converge the probability of failure strategy to 0.01. The defender has obtained the optimal defense strategy through several stages of learning. The game defense is successful.


## 6.5. Multistrategies Failure Scene

In status III, the attacker can choose any attack strategy to make any defense strategy invalid. In this scene, the defense strategy that failed may beDS1 in stage K, and the defense strategy that failed may become DS2 in stage K+1. So, the game becomes more complex.Figure5 shows the defense strategy selection probability in status III. As shown in Figure 5, the change of defense strategy selection probability is unstable, which is caused by the high complexity of the network state in this scene. In this scene, the defense side tends to give a more selection probability to the defense strategy DS1.In status III, the failed strategies of each stage are not necessarily the same, so we cannot judge the model from the probability of defense strategy selection experiment. However, the model can be verified from attack and defense revenue experiment and defense success rate experiment in status III.


## 6.6. Attack and Defense Revenue Experiment

Figures6(a), 6(b), and 7 show the comparative diagram of attack and defense revenue values based on our model and the evolutionary game model without RLM in statuses I, II, and III. The x-axis and y-axis are game stage and attack and defend revenue. The blue line and the yellow line represent the defense and the attack revenue of our model. The green line and the red line represent the defense and the attack revenue of the evolutionary game model without RLM.Figure 6
Comparison of attack and defense revenue under status I and II. (a) Status I. (b) Status II.
(a)(b)Figure 7
Comparison of attack and defense revenue under status III.


## 6.7. Single Strategy Failure Scene

Figures6(a) and 6(b) show the comparison of the attack and defense revenue values of the game model in the number of stages K=30 when the status is I and II.It can be seen from Figures6(a) and 6(b) that the difference between the attack and defense revenues of the two models is not obvious at the beginning of the game stage, which indicates that RLM has not fully learned the attacker’s information. At this time, the attack and defense revenues of the two models are unstable. Even because of the change of attack strategy, the revenues of the attacker may become more, and the revenues of the defender may become less.After several stages, the defense revenues of the two models are positively correlated with the game stage, and the attack revenues are negatively correlated with the game stage. Compared with the game model without RLM, the model in Figure6(a) has higher defense revenue and lower attack revenue after stage K=10, and the model in Figure 6(b) has lower attack revenue after stage K=15 and higher defense revenue after stage K=20. Combined with the previous analysis, we can conclude that the revenue value in the game model with RLM changes drastically from the probability convergence of defense strategy. This situation also means that after the short-term learning stages of RLM, the defense revenue begins to rise, and the attack revenue begins to decline.Compared with Figure6(a), the change of attack and defense revenue in Figure 6(b) is relatively slow because any attacker’s strategy may make defense strategy DS1 invalid when the status is II. To sum up, this model can more effectively resist the attackers in the scenes of statuses I and II.


## 6.8. Multistrategies Failure Scene

Figure7 shows the comparison of attack and defense revenue between our game model and the game model without RLM when the status is III. From the previous description, we can conclude that the scene is highly complex, and the attacker can adjust the strategy at each stage to invalidate the specific defense strategy. Therefore, with the continuous change of attack strategy, the attack and defense revenues fluctuate. Next, from the perspective of attack and defense revenues, we verify that our model has better defense ability than the model without RLM in status III.As can be seen from Figure7, the defense revenue of our model and the attack revenue of the game model without RLM rise with fluctuation, and the attack revenue of our model and the defense revenue of the game model without RLM decline with fluctuation. From stage K=400, the defense revenue of our model is always higher than that of the game model without RLM. From stage K=300, the attack revenue of this model is always lower than that of the game model without RLM.To sum up, after learning multiple stages with RLM, the model can make the defense revenue greater and the attack revenue lower, to better resist network attacks.


## 6.9. Defense Success Rate Experiment

Figure8 shows the comparison of defense success rate between our model and the game model without RLM in stage K=1000. The green column denotes the number of successful defense stages when the models choose defense strategy DS1. The blue column denotes the number of successful defense stages when the models choose defense strategy DS2. The red column denotes the number of failed defense stages when the models choose defense strategy DS1. The yellow column denotes the number of failed defense stages when the models choose defense strategy DS2.Figure 8
Comparison of defense success rate.It can be seen from Figure8 that our model under statuses I, II, and III has a higher defense success rate than the game model without RLM. Compared with the game model without RLM, the defense success rate of this model is about 22.5% higher under state I. The defense success rate of our model increases by about 23.5% under status II. Our model improves by about 7.4% under status III. In conclusion, compared with the evolutionary game model without RLM, our model can better select the best defense strategy in the above three scenes.


## 7. Conclusion

Considering the existing problems in the application of the evolutionary game model in network attack and defense, this paper proposes a reward value learning mechanism. This mechanism overcomes the problem of quantifying incentives and punishments in the case of bounded rationality of attackers and defenders, which reduces manual involvement. An evolutionary game model with a multistage learning mechanism is constructed by combining the learning mechanism with a multistage game model. Furthermore, the optimal strategy selection algorithm of the game model is designed.Our future work will study how to dynamically add new feasible defense strategies and reasonably expand the model when any defense strategy fails. In addition, we will also consider how to apply more intelligent methods, such as deep learning and machine learning, to the automatic calculation of reward and punishment factorα at every stage so that the model can better select the optimal defense strategy.



---

*Source: 4773894-2021-11-11.xml*