# Hierarchical Q-Learning Based UAV Secure Communication against Multiple UAV Adaptive Eavesdroppers

Authors: Jue Liu; Nan Sha; Weiwei Yang; Jia Tu; Lianxin Yang
Journal: Wireless Communications and Mobile Computing
 (2020)

---

## Abstract

In this paper, we investigate secure unmanned aerial vehicle (UAV) communication in the presence of multiple UAV adaptive eavesdroppers (AEs), where each AE can conduct eavesdropping or jamming adaptively by learning others’ actions for degrading the secrecy rate more seriously. The one-leader and multi-follower Stackelberg game is adopted to analyze the mutual interference among multiple AEs, and the optimal transmit powers are proven to exist under the existing conditions. Following that, a mixed-strategy Stackelberg Equilibrium based on finite and discretized power set is also derived and a hierarchical Q-learning based power allocation algorithm (HQLA) is proposed to obtain the optimal power allocation strategy of the transmitter. Numerical results show that secrecy performance can be degraded severely by multiple AEs and verify the availability of the optimal power allocation strategy. Finally, the effect of the eavesdropping cost on the AE’s attack mode strategies is also revealed.


---

## Body

## 1. Introduction

With the inherent advantages in mobility, flexibility, and adaptive altitude, unmanned aerial vehicle (UAV) wireless communication has experienced an upsurge of interest in both military and civilian applications [1–6]. However, both the broadcast nature of the wireless medium and the malicious attackers make the electromagnetic environment of UAV communication hostile. Hence, the security issue of UAV communications is of paramount importance yet a significant challenge [7].As an option, the physical layer security (PLS) technique with lower computation complexity has been proven that it can protect wireless communication networks from wiretapping and interfering by exploiting the random characteristics of wireless channels in recent years [8–11]. Naturally, due to the payload-limited characteristic of UAV, many PLS approaches [12–23] combining with the high altitude and mobility of UAV have been applied widely in UAV-involved communications.However, most approaches in the above work that mainly focused on the single-mode scenarios are not fully suitable for the novel attackers, named as “adaptive eavesdroppers (AEs),” “active eavesdropper,” or “smart eavesdropper.” They use programmable radio devices to flexibly choose their attack methods, such as eavesdropping, jamming, and spoofing, according to the ongoing transmission status and the radio channel states. For example, an AE sends spoofing signals if she has a similar channel state with Alice or sends jamming signals if she is very close to Bob. Compared with the traditional single-mode attackers each performing a single-mode attack, an AE can be more harmful to the UAV transmission by reducing the secrecy capacity. Therefore, it is urgent to investigate the effective countermeasures against this type of eavesdropper.In recent years, some literature began to investigate AE. One form of the AE is achieved by the manner of multiantenna full-duplex (FD) technology [24–27], which can assign one part of antennas to wiretap and the other antennas interfere simultaneously. Another type of AE emerges during the channel estimation phase in the form of time-division duplex, and it leads to pilot contamination by sending the same pilot sequence as the legitimate node [28–31]. While in the data transmission phase, the AE reverts to passive eavesdropper again. Nevertheless, it should be noticed that the attack modes of these two forms of AE are predefined which means it cannot change the attack mode adaptively. The third form of AE in [32–36] can adjust its attack strategies adaptively. But there are still several problems remaining unsolved. Firstly, current work rarely considered multiple AEs case and the mutual interference between themselves. Secondly, existing studies neglected the AE adaptivity supported by the learning ability in searching for the optimal strategies of the transmitter and did not reveal the impact of the learning ability of AE on the secrecy performance of the considered system. How to search the transmitter’s optimal power strategies in the face of multiple AEs with the learning ability and how to handle the mutual interference from AEs to improve the security capacity of the UAV communication system are necessary to be considered.In our work, we mainly concentrate on a secure UAV communication scenario in the presence of multiple UAV AEs, which can eavesdrop or jam adaptively by learning others’ strategies as well as dynamic environments. For the considered scenario, each AE’s attack activity may affect the signal to interference plus noise ratio (SINR) of others. This implies that each AE’s decision-making is not only coupled with the interactions from the transmitter but also from other AEs. Considering these hierarchical interactions between the transmitter-side and AEs-side, the Stackelberg game [37–41] is a suitable framework to capture the sequential interactions between the transmitter and AEs. Then, the Stackelberg Equilibrium (SE) points of the formulated game turn to be the feasible solutions to the transmit power allocation problem. However, the SE points solely provide theoretic solutions and it is challenging to obtain the SE solutions. In particular, the AE with the learning ability in this paper makes decisions spontaneously and independently, which results in unpredictable attack modes of the whole AE set. In this context, it is not feasible to handle this problem by centralized means because the number of each attack mode and locations of AEs are unknown, which motivates applying the idea of reinforcement learning (RL). So, we incorporate RL technology into the proposed game and a hierarchical Q-learning based power allocation algorithm is proposed to obtain the mixed-strategy equilibrium solution. The main contributions of this paper are summarized as follows:
(i)
We propose a secure UAV communication model which constitutes of one transmitter-receiver pair and multiple UAV AEs. Each AE decides to eavesdrop or jam adaptively by learning the other nodes’ strategies as well as the dynamic environment to maximize its damage. Also, the interference among AEs is investigated.(ii)
We formulate the UAV secure transmission problem as a one-leader and multi-follower Stackelberg game where the transmitter acts as the leader and all AEs are followers. The optimal transmit power of leader are obtained by analyzing the pure strategy SEs under the existing conditions. Besides, the mixed-strategy SE is also derived for the finite and discretized power set. Then, we apply a hierarchical RL framework in which each player chooses its attack strategy based on a probability distribution and a hierarchical Q-learning based power allocation algorithm is proposed to discover the mixed-strategy equilibrium of the formulated game. Besides, we provide rigorous theoretical proof about the convergence of the proposed algorithm.(iii)
Numerical results show the availability of the optimal power allocation strategy of the legitimate transmitter in the more hostile situation and reveal the impact of AE’s learning ability on the secrecy rate. Meanwhile, we show that the proposed algorithm has a significant convergence advantage over the single-agent RL algorithm. Finally, the effect of the eavesdropping cost on the AE’s attack mode strategies is also revealed.(iv)
We organize the rest of this paper as follows. In Section2, we present the related work. Then, we present the system model in Section 3. In Section 4, we formulate the UAV secure transmission game and investigate a power allocation policy in Section 5. In Section 6, we provide the simulation results and conclude the work in Section 7.


## 2. Related Work

In UAV communication, there have been abundant approaches, such as 3D beamforming [12–14], trajectory optimization [5, 15–19], multi-UAV cooperation [17, 20], and resource management techniques [21–23], concerning on the single attack mode. Whereas, it is inappropriate to apply them directly to defend the novel attacker that has the multiple abilities of eavesdropping, jamming, spoofing, and so on.As a novel attacker, the AE can eavesdrop and jam simultaneously by the FD capability [24–27]. Specifically, Tang et al. investigated the physical layer security issue in the presence of an FD AE within a hierarchical game framework in [24]. In [25], Mukherjee and Swindlehurst examined the design of an FD active eavesdropper in the 3-user MIMOME wiretap channel, where the adversary intends to optimize its transmit and receive sub-arrays and jamming signal parameters to minimize the MIMO secrecy rate of the main channel. In [26], the potential benefits of an FD receiver node in the presence of an active FD eavesdropper was studied. The optimal receive/transmit antennas allocation at the receiver against active eavesdropper in an FD pattern is provided in [27]. The second AE scenario adopts time-division duplex technology. The adaptive eavesdropper sent the same pilot sequence as the legitimate user node in the training phase leading to pilot contamination [28–31]. Zhou et al. discussed how an AE attacked the training phase in wireless communication to improve its eavesdropping performance in [28]. A simple protocol to determine whether an AE is present or not using the channel properties of MMIMO is proposed in [29]. A novel random-training-assisted (RTA) pilot spoofing detection algorithm and a zero-forcing based secure transmission scheme is proposed to protect the confidential information from the active eavesdropper in [30]. Unfortunately, all AEs in the above scenarios cannot adjust attack mode adaptively. More recently, the AE that can determine the attack mode autonomously has been studied in [32–36]. To be specific, Li et al. studied the secure communication game under the AE from UAV with the imperfect channel estimation but ignored the mobility of UAV in [32]. Li et al. formulated the MIMO transmission in the presence of AE as a noncooperative game and obtained the power control strategy based on Q-learning in [33]. Zhu et al. proposed a noncooperative strategic game to make a complex decision between users that perform uplink transmission via relay and an active malicious node in [34]. In [35], Xiao et al. formulated a subjective smart attack game for the UAV transmission and proposed a deep Q-learning RL based UAV power allocation strategies. However, these above researches did not refer to the multiple AEs’ scenario, and the mutual interference between AEs is hardly considered. Moreover, these AEs cannot learn from others’ strategies and the dynamic environment. A summary of the proposed literature about AE has been given in Table 1.Table 1
A summary of the proposed literature about AE.
Ref.
FD/HD
Attacker’s antennas
Attack mode and phase
Attacker number
Solution
Advantages
Disadvantages
Theory
Algorithm
24 (2016)
FD
Multiple antennas
Eavesdrop and jam during data transmission
1
Stackelberg game
Primal-dual interior-point method
Consider the physical layer security issue for multi-channel wireless communications in the presence of an FD active eavesdropper.
Not adaptive
25 (2011)
FD
Multiple antennas
Eavesdrop and jam during data transmission
1
Gradient projection + fixed-point iteration algorithm
Solve the worst-case jamming covariance for arbitrary and Gaussian input signaling.
Not adaptive
26 (2017)
FD
Multiple antennas
Eavesdrop and jam during data transmission
1
Convex optimization
Primal decomposition iterative algorithm
The channel state information is uncertain.
Not adaptive
27 (2017)
HD
Multiple antennas
Eavesdrop and jam during data transmission
1
Constructing the precoding matrix pair at Alice and Bob.
Optimal antennas allocation of the receiver to maximize the achievable secrecy degrees of freedom.
Not adaptive
28 (2012)
HD
Single antenna
Jam during pilot training and eavesdrop during data transmission
1
Convex optimization
Introduce the active eavesdropper into the pilot contamination phenomenon.
Not adaptive
30 (2017)
HD
Single antenna
Jam during pilot training and eavesdrop during data transmission
1
Random-training-assisted spoofing detection scheme
Adding a random training phase after the conventional pilot training phase.
Not adaptive
32 (2018)
HD
Single antenna
Silence, spoof, eavesdrop, and jam during data transmission
1
Stackelberg game
Single-agent Q-learning algorithm
Solve the physical layer security issue under imperfect channel estimation.
Single-agent Q-learning
33 (2017)
FD
Multiple antennas
Eavesdrop, jam, spoof, or keep silent during data transmission
1
Noncooperative game
Single-agent Q-learning algorithm
Solve the physical layer security issue under imperfect channel estimation.
Not adaptive
34 (2011)
HD
Single antenna
Eavesdrop and jam during data transmission
1
Mixed-strategy based noncooperative nonzero-sum game
Fictitious play-based iterative algorithm
Reveal the impact of the presence of a malicious node on the multi-relay to multi-user choices.
Adaptive
35 (2017)
HD
Single antenna
Eavesdrop, jam, and spoof during data transmission
1
PT-based dynamic smart attack game
Q-learning\WoLF-PHC\DQN algorithms
Apply the prospect theory.
AdaptiveOur work in this paper is different from the above researches that we focus on the AE with learning ability that can choose the attack mode independently and investigate the secure transmission problem of UAV communication in the presence of multiple AEs. Note that the approach of defending multiple AEs using the Stackelberg game in UAV communication networks was presented in our previous work [37], and the main differences and new contributions are (i) aim to the actual UAV communication, we introduce the mixed-strategies for the discretized transmit power set, and (ii) we assume that each AE has the learning ability and reveal the impact of the AE’s learning ability on the secrecy rate. Besides, the similarity between the most related work in [32] and our work is that the Stackelberg game-based power allocation problem in the secure transmission of UAV communication is investigated. The main differences are (i) we consider the multiple AEs case which is more actual in UAV communication while the work in [32] ignores it, and (ii) the mutual interference among themselves is considered.


## 3. System Model

As shown in Figure1, we consider the downlink of a UAV communication system consisting of a transmitter (Alice), a receiver (Bob), and M number of UAV AEs randomly distributed around transmitter-receiver pairs, where all nodes are single-antenna and UAVs are all hovering. Here, we adopt a 3D Cartesian coordinate system with the Alice, Bob, and theAEm located at xa,ya,ha, xb,yb,hb, and xm,ym,hm. Alice communicates with Bob by using transmit power that is denoted by Ps∈0,Pmax, where Pmax is the maximum transmit power. Without the loss of generality, being a programmable radio device, when Alice is transmitting a signal to Bob, some AEs act as passive eavesdroppers to overhear Alice’s signals if they can derive enough information. The rest of the AEs send jamming signals if they can effectively block Alice’s signal to Bob. Each AE can either eavesdrop on Alice or jam Bob, under a half-duplex constraint. Here qm∈e,j,m∈1,M, corresponding to eavesdropping and jamming, denotes the specific attack mode of AEm. Hence, the sets of the passive eavesdroppers and the active jammers can be denoted by ΦE and ΦJ, respectively, where ∣ΦJ∣+ΦE=M.Figure 1
System Model.Considering the low mobility of low-altitude UAVs, all the channels are assumed to be quasi-static fading, i.e., the channel gains are constant with each transmission block. Besides, the channel gains between the UAVs follow the free-space path loss model, which is determined by the distance between the UAVs, i.e.,(1)gi,j=β0di,j−η=β0ζi−ζj2η,where β0 is the channel power gain at the reference distance of d0=1m, di,j is the distance from node i to node j, ζi is the coordinate of node i, and η is the path loss exponent.Remark 1.
Since each AE can eavesdrop or interfere adaptively by learning the communication environment, Alice and other AEs can monitor the AE’s position when it chooses to jam. So, we assume that the number and locations of all nodes (legitimate communication pairs and all AEs) are available between each other via a priori measurement following the above analysis. In addition, as the AE considered in this paper can only eavesdrop and interfere, at each time slot, each node judged other’s actions by sensing the jamming signal. If one AE does not jam, other nodes consider that it chooses to eavesdrop.At each time slot, Alice first sends a normalized signalxa with transmit power Ps. Then, all AEs conduct different attack modes by learning others’ strategies. The legitimate link and all passive eavesdroppers suffer interference from all active jammers. The interference to legitimate link and the kth passive eavesdropper (k∈ΦE) is given by ∑j∈ΦJPjgj,b and ∑j∈ΦJPjgj,k, where Pj is the jamming power.The received signal at Bob can be expressed as(2)yb=Psga,bxa+∑j∈ΦJPjgj,bxj+nb,where nb~CN0,σn2 is the additive white Gaussian noise (AWGN) at Bob. The received SINR at Bob can be expressed as
(3)rab=Psω0da,b−ηIJ,B+1,where ω0=β0/σn2 and IJ,B=∑j∈ΦJPjω0dj,b−η that denotes the interference from all AEs who choose to jam. We can obtain the data rate of the Alice-Bob link as
(4)Rab=log21+rab.Due to Remark1, each AE can get the other AEs’ actions. So, the signal received at the kth passive eavesdropper can be expressed as
(5)ye=Psga,kxa+∑j∈ΦJPjgj,kxj+ne,where ne~CN0,σn2 is the AWGN at the kth passive eavesdropper. Similarly, the received SINR at the kth passive eavesdropper can be expressed a
(6)rk=Psω0da,k−ηIJ,E+1,where IJ,E=∑j∈ΦJPjω0dj,k−η.Assuming the maximal eavesdropped information is determined by the maximal SINR among all passive eavesdroppers, i.e.,rE=maxk∈ΦErk. We obtain the maximal data rate of the Alice-AE links, which is given as
(7)Rae=log21+rE.From (4) and (7), the secrecy rate of Alice can be written as
(8)Ra=Rab−Rae+,where X+ returns X if X is positive, while returns 0 otherwise.


## 4. Secure Transmission Game

In this section, we investigate the secure transmission problem with multiple UAV AEs. The interactions between the transmitter and multiple UAV AEs are formulated under the Stackelberg game framework. The optimal power allocations and secrecy rate of Alice and the best attack modes of all AEs are derived by analyzing the equilibrium of the game.


### 4.1. Secure Transmission Game Formulation

The secure transmission problem of this proposed system can be formulated as a two-stage Stackelberg game. Specifically, Alice is a leader and all AEs are followers. Alice decides its transmit power firstly and all AEs take their action adaptively based on the observation of the leader’s action in the sequel. The secure transmission game is formulated as(9)G=N,P,Q,Ua,Um.Here,N=Alice,AE1,⋯,AEm,⋯,AEM is modeled as the players, and P∈0,Pmax and Q=e,j are the strategy space of Alice and AE, respectively. Also, Ua and Um are the utility of Alice and AE, respectively.In this system, Alice wants to send a confidential message and thus naturally intends to maximize its secrecy rate. Meanwhile, the transmission cost is inevitable during the transmission. Therefore, the utility of the leader is the trade-off of the secrecy rate and transmission cost, which can be formulated as(10)Ua=Raln2−CaPs,where Ca denotes the cost of the unit transmit power of Alice. For computational convenience, we multiply the data rate by a coefficient ln2.The objective of the leader is to solve the following problem to obtain the optimal power allocation:(11)Ps∗=argmaxPs∈0,PmaxUaPs,qm∗,q−m∗,where q1∗,⋯,qm∗ denotes the optimal action of all AEs.On the other hand, each AE attempts to minimize the secrecy rate of Alice by changing its attack mode adaptively according to Alice’s transmit power. Therefore, we formulate the utility ofAEm with the trade-off of the secrecy rate and its attack cost as follows
(12)Um=−Ra−θqm,θqm∈e,j,where θe and θj denotes the cost of each AE to perform as the passive eavesdropper and active jammer, respectively. We assume that θe is related to the Rae, i.e., θe=CeRae, where Ce denotes the cost of unit rate of Rae. θj=CjPj, where Cj denotes the cost of the unit transmit power of jammer.To calculate the utility of a single AE accurately, at each time slot, when Alice is transmitting a signal to Bob, we divide all AEs into three parts, which are denoted asΦE−m, ΦJ−m, and AEm, respectively, i.e., ∣ΦE−m∣+∣ΦJ−m∣+AEm=M. ΦE−m is the set of passive eavesdroppers except AEm and ΦJ−m is the set of active jammers except AEm.IfAEm decides to act as a passive eavesdropper, the Ra can be expressed as
(13)Ra=Rab−Rae+=log21+rab−log21+rE+=log21+Psω0da,b−ηIJ,B−m+1−log21+maxmaxk∈ΦE−mrk,rm+=log21+Psω0da,b−ηIJ,B−m+1−log21+maxrE−m,rm+,where IJ,B−m is the interference received at Bob from ΦJ−m, and rm is the SINR of AEm.Similarly, ifAEm selects to jam, Ra can be expressed as
(14)Ra=Rab−Rae+=log21+rab−log21+rE+=log21+Psω0da,b−ηIJ,B−m+Im+1−log21+maxk∈ΦE−mrk+,where Im is the jamming power of AEm, and rE−m is the maximal SINR among all passive eavesdroppers in ΦE−m.In conclusion,Um can be expressed as
(15)Um=log21+Psω0da,b−ηIJ,B−m+1−log21+maxrE−m,rm+−CeRae,qm=ea,log21+Psω0da,b−ηIJ,B−m+Im+1−log21+maxk∈ΦE−mrk+−CjPj,qm=jb.Similarly, the objective ofAEm is to solve the following problem:
(16)qm∗=argmaxqm∈e,jUmPs∗,qm,q−m∗,where q−m∗ denotes the optimal action of all AEs except AEm.


### 4.2. Analysis of Strategy Equilibrium

Now, we will analyze the proposed Stackelberg game model and solve the optimization subproblems of (11) and (16). As a follower, each AE will adjust its attack mode after sensing Alice’s strategy. Therefore, the subgame of followers is analyzed firstly.Proposition 1.
Given the strategy of Alice, the optimal attack mode strategy ofAEm is expressed as (17) if (17(a)) and (17(b)) hold.
(17)qm∗Ps=e,ifCjPj≥log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Cea,j,ifCjPj≤log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Ceb,where Ps∗ is the optimal power allocation, and IJ,B−m∗=∑j∈ΦJ−m∗Pjω0dj,b−η denotes the interference from ΦJ−m∗, in which each AE chooses to jam as an optimal strategy, rE−m∗=maxk∈ΦE−m∗rk denotes the maximal SINR among all AEs in ΦE−m∗ where each AE chooses to overhear as an optimal strategy.Proof.
If (17(a)) holds, from (12), we have
(18)UmPs∗,e,q−m∗−UmPs∗,j,q−m∗=−log21+Ps∗ω0da,b−ηIJ,B−m∗+1−log21+maxrE−m∗,rm+log21+Psω0da,b−ηIJ,B−m∗+Im+1−log21+rE−m∗−Celog21+maxrE−m∗,rm+CjPj=−log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Ce+CjPj≥0.If (17(b)) holds, from (12), we have
(19)UmPs∗,j,q−m∗−UmPs∗,e,q−m∗=log21+Ps∗ω0da,b−ηIJ,B−m∗+1−log21+maxrE−m∗,rm−log21+Psω0da,b−ηIJ,B−m∗+Im+1−log21+rE−m∗+Celog21+maxrE−m∗,rm−CjPj=log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Ce−CjPj≥0.Thus (17) holds.As shown in Proposition1, if passive eavesdropping can bring worse secrecy rate and less cost than active jamming, the AE will select to overhear and vice versa.As the leader of the game, Alice first chooses to transmit power. The optimal power strategy of Alice can be derived by solving (11), which is revealed in Proposition 2.Proposition 2.
The optimal power allocation isPs∗, which satisfies the following equation:
(20)ω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+1IJ,B−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+IJ,B∗+Ps∗ω0da,b−η1+Ps∗ω0maxk∈ΦE∗da,k−η/IJ,E∗+1=Caa,0≤Ps∗≤Pmaxb,if (21(a)) and (21(b)) hold.
(21)maxk∈ΦE∗da,k−ηIJ,E∗+1<da,b−ηIJ,B∗+1a,ω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+1IJ,B−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+IJ,B∗+Ps∗ω0da,b−η1+Ps∗ω0maxk∈ΦE∗da,k−η/IJ,E∗+1≤Ca≤ω0da,b−η1+IJ,B∗−ω0maxk∈ΦE∗da,k−ηIJ,E∗+1b,where IJ,B∗ and IJ,E∗ denotes the interference from ΦJ∗ in which each AE chooses to jam as an optimal strategy to Bob and the kth passive eavesdropper, respectively.Proof.
We obtain the following differential equation describing the evolution of the utility of Alice:(22)∂Ua∂Ps=ω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+1IJ,B−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+IJ,B∗+Ps∗ω0da,b−η1+Ps∗ω0maxk∈ΦE∗da,k−η/IJ,E∗+1−Ca,(23)∂2Ua∂Ps2=ω0da,b−η1+IJ,B∗+Psω0da,b−η+ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Psω0maxk∈ΦE∗da,k−η/IJ,E∗+1ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Psω0maxk∈ΦE∗da,k−η/IJ,E∗+1−ω0da,b−η1+IJ,B∗+Psω0da,b−η.
If (21(a)) holds, (23) is less than zero. Thus, we have
(24)∂2Ua∂Ps2<0,which indicates that ∂Ua/∂Ps monotonically decreases with Ps. Therefore, if (21(b)) holds, we have
(25)∂Ua∂PsPs=0=ω0da,b−η1+IJ,B∗−ω0maxk∈ΦE∗da,k−ηIJ,E∗+1−Ca>0,(26)∂2Ua∂Ps2Ps=Pmax=ω0da,b−η1+IJ,B∗+Pmaxω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Pmaxω0maxk∈ΦE∗da,k−η/IJ,E∗+1−Ca<0,indicating that there is a sole solution to ∂Ua/∂Ps=0, given in (20(a)). From (22)–(24), we can find that UaPs,qm∗,q−m∗ increases with Ps, if Ps<Ps∗, while it decreases otherwise. Thus, (11) also holds and Ps,qm∗,q−m∗ is a Nash Equilibrium (NE) of the game. In this way, we have completed the proof of Proposition 2.As shown in Proposition2, Alice stops the transmission when (21(b)) does not hold. In other words, Alice will stop the transmission under the circumstances that radio channel degradation is serious and the security cannot be guaranteed.Another NEPmax,qm∗,q−m∗ is revealed in Proposition 3.Proposition 3.
The secure game has the NEPmax,qm∗,q−m∗ if (21(a)) and the following equation hold:
(27)ω0da,b−η1+IJ,B∗+Pmaxω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Pmaxω0maxk∈ΦE∗da,k−η/IJ,E∗+1>Ca.Proof.
(21(a)) has been discussed above.
Therefore, if (27) holds, we have
(28)∂Ua∂Ps≥∂Ua∂PsPs=Pmax≥0,∀0≤Ps≤Pmax,which indicates that Ua monotonically increases with Ps, Pmax,qm∗,q−m∗ is also an NE of the game. In this way, we have completed the proof of Proposition 3.As shown in Proposition3, low transmission costs in (27) will make Alice select the maximum transmit power to transmit the signals.


## 4.1. Secure Transmission Game Formulation

The secure transmission problem of this proposed system can be formulated as a two-stage Stackelberg game. Specifically, Alice is a leader and all AEs are followers. Alice decides its transmit power firstly and all AEs take their action adaptively based on the observation of the leader’s action in the sequel. The secure transmission game is formulated as(9)G=N,P,Q,Ua,Um.Here,N=Alice,AE1,⋯,AEm,⋯,AEM is modeled as the players, and P∈0,Pmax and Q=e,j are the strategy space of Alice and AE, respectively. Also, Ua and Um are the utility of Alice and AE, respectively.In this system, Alice wants to send a confidential message and thus naturally intends to maximize its secrecy rate. Meanwhile, the transmission cost is inevitable during the transmission. Therefore, the utility of the leader is the trade-off of the secrecy rate and transmission cost, which can be formulated as(10)Ua=Raln2−CaPs,where Ca denotes the cost of the unit transmit power of Alice. For computational convenience, we multiply the data rate by a coefficient ln2.The objective of the leader is to solve the following problem to obtain the optimal power allocation:(11)Ps∗=argmaxPs∈0,PmaxUaPs,qm∗,q−m∗,where q1∗,⋯,qm∗ denotes the optimal action of all AEs.On the other hand, each AE attempts to minimize the secrecy rate of Alice by changing its attack mode adaptively according to Alice’s transmit power. Therefore, we formulate the utility ofAEm with the trade-off of the secrecy rate and its attack cost as follows
(12)Um=−Ra−θqm,θqm∈e,j,where θe and θj denotes the cost of each AE to perform as the passive eavesdropper and active jammer, respectively. We assume that θe is related to the Rae, i.e., θe=CeRae, where Ce denotes the cost of unit rate of Rae. θj=CjPj, where Cj denotes the cost of the unit transmit power of jammer.To calculate the utility of a single AE accurately, at each time slot, when Alice is transmitting a signal to Bob, we divide all AEs into three parts, which are denoted asΦE−m, ΦJ−m, and AEm, respectively, i.e., ∣ΦE−m∣+∣ΦJ−m∣+AEm=M. ΦE−m is the set of passive eavesdroppers except AEm and ΦJ−m is the set of active jammers except AEm.IfAEm decides to act as a passive eavesdropper, the Ra can be expressed as
(13)Ra=Rab−Rae+=log21+rab−log21+rE+=log21+Psω0da,b−ηIJ,B−m+1−log21+maxmaxk∈ΦE−mrk,rm+=log21+Psω0da,b−ηIJ,B−m+1−log21+maxrE−m,rm+,where IJ,B−m is the interference received at Bob from ΦJ−m, and rm is the SINR of AEm.Similarly, ifAEm selects to jam, Ra can be expressed as
(14)Ra=Rab−Rae+=log21+rab−log21+rE+=log21+Psω0da,b−ηIJ,B−m+Im+1−log21+maxk∈ΦE−mrk+,where Im is the jamming power of AEm, and rE−m is the maximal SINR among all passive eavesdroppers in ΦE−m.In conclusion,Um can be expressed as
(15)Um=log21+Psω0da,b−ηIJ,B−m+1−log21+maxrE−m,rm+−CeRae,qm=ea,log21+Psω0da,b−ηIJ,B−m+Im+1−log21+maxk∈ΦE−mrk+−CjPj,qm=jb.Similarly, the objective ofAEm is to solve the following problem:
(16)qm∗=argmaxqm∈e,jUmPs∗,qm,q−m∗,where q−m∗ denotes the optimal action of all AEs except AEm.


## 4.2. Analysis of Strategy Equilibrium

Now, we will analyze the proposed Stackelberg game model and solve the optimization subproblems of (11) and (16). As a follower, each AE will adjust its attack mode after sensing Alice’s strategy. Therefore, the subgame of followers is analyzed firstly.Proposition 1.
Given the strategy of Alice, the optimal attack mode strategy ofAEm is expressed as (17) if (17(a)) and (17(b)) hold.
(17)qm∗Ps=e,ifCjPj≥log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Cea,j,ifCjPj≤log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Ceb,where Ps∗ is the optimal power allocation, and IJ,B−m∗=∑j∈ΦJ−m∗Pjω0dj,b−η denotes the interference from ΦJ−m∗, in which each AE chooses to jam as an optimal strategy, rE−m∗=maxk∈ΦE−m∗rk denotes the maximal SINR among all AEs in ΦE−m∗ where each AE chooses to overhear as an optimal strategy.Proof.
If (17(a)) holds, from (12), we have
(18)UmPs∗,e,q−m∗−UmPs∗,j,q−m∗=−log21+Ps∗ω0da,b−ηIJ,B−m∗+1−log21+maxrE−m∗,rm+log21+Psω0da,b−ηIJ,B−m∗+Im+1−log21+rE−m∗−Celog21+maxrE−m∗,rm+CjPj=−log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Ce+CjPj≥0.If (17(b)) holds, from (12), we have
(19)UmPs∗,j,q−m∗−UmPs∗,e,q−m∗=log21+Ps∗ω0da,b−ηIJ,B−m∗+1−log21+maxrE−m∗,rm−log21+Psω0da,b−ηIJ,B−m∗+Im+1−log21+rE−m∗+Celog21+maxrE−m∗,rm−CjPj=log21+Ps∗ω0da,b−η+IJ,B−m∗IJ,B−m∗+Im+11+rE−m∗1+Ps∗ω0da,b−η+IJ,B−m∗+ImIJ,B−m∗+11+maxrE−m∗,rm1−Ce−CjPj≥0.Thus (17) holds.As shown in Proposition1, if passive eavesdropping can bring worse secrecy rate and less cost than active jamming, the AE will select to overhear and vice versa.As the leader of the game, Alice first chooses to transmit power. The optimal power strategy of Alice can be derived by solving (11), which is revealed in Proposition 2.Proposition 2.
The optimal power allocation isPs∗, which satisfies the following equation:
(20)ω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+1IJ,B−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+IJ,B∗+Ps∗ω0da,b−η1+Ps∗ω0maxk∈ΦE∗da,k−η/IJ,E∗+1=Caa,0≤Ps∗≤Pmaxb,if (21(a)) and (21(b)) hold.
(21)maxk∈ΦE∗da,k−ηIJ,E∗+1<da,b−ηIJ,B∗+1a,ω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+1IJ,B−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+IJ,B∗+Ps∗ω0da,b−η1+Ps∗ω0maxk∈ΦE∗da,k−η/IJ,E∗+1≤Ca≤ω0da,b−η1+IJ,B∗−ω0maxk∈ΦE∗da,k−ηIJ,E∗+1b,where IJ,B∗ and IJ,E∗ denotes the interference from ΦJ∗ in which each AE chooses to jam as an optimal strategy to Bob and the kth passive eavesdropper, respectively.Proof.
We obtain the following differential equation describing the evolution of the utility of Alice:(22)∂Ua∂Ps=ω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+1IJ,B−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+IJ,B∗+Ps∗ω0da,b−η1+Ps∗ω0maxk∈ΦE∗da,k−η/IJ,E∗+1−Ca,(23)∂2Ua∂Ps2=ω0da,b−η1+IJ,B∗+Psω0da,b−η+ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Psω0maxk∈ΦE∗da,k−η/IJ,E∗+1ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Psω0maxk∈ΦE∗da,k−η/IJ,E∗+1−ω0da,b−η1+IJ,B∗+Psω0da,b−η.
If (21(a)) holds, (23) is less than zero. Thus, we have
(24)∂2Ua∂Ps2<0,which indicates that ∂Ua/∂Ps monotonically decreases with Ps. Therefore, if (21(b)) holds, we have
(25)∂Ua∂PsPs=0=ω0da,b−η1+IJ,B∗−ω0maxk∈ΦE∗da,k−ηIJ,E∗+1−Ca>0,(26)∂2Ua∂Ps2Ps=Pmax=ω0da,b−η1+IJ,B∗+Pmaxω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Pmaxω0maxk∈ΦE∗da,k−η/IJ,E∗+1−Ca<0,indicating that there is a sole solution to ∂Ua/∂Ps=0, given in (20(a)). From (22)–(24), we can find that UaPs,qm∗,q−m∗ increases with Ps, if Ps<Ps∗, while it decreases otherwise. Thus, (11) also holds and Ps,qm∗,q−m∗ is a Nash Equilibrium (NE) of the game. In this way, we have completed the proof of Proposition 2.As shown in Proposition2, Alice stops the transmission when (21(b)) does not hold. In other words, Alice will stop the transmission under the circumstances that radio channel degradation is serious and the security cannot be guaranteed.Another NEPmax,qm∗,q−m∗ is revealed in Proposition 3.Proposition 3.
The secure game has the NEPmax,qm∗,q−m∗ if (21(a)) and the following equation hold:
(27)ω0da,b−η1+IJ,B∗+Pmaxω0da,b−η−ω0maxk∈ΦE∗da,k−η/IJ,E∗+11+Pmaxω0maxk∈ΦE∗da,k−η/IJ,E∗+1>Ca.Proof.
(21(a)) has been discussed above.
Therefore, if (27) holds, we have
(28)∂Ua∂Ps≥∂Ua∂PsPs=Pmax≥0,∀0≤Ps≤Pmax,which indicates that Ua monotonically increases with Ps, Pmax,qm∗,q−m∗ is also an NE of the game. In this way, we have completed the proof of Proposition 3.As shown in Proposition3, low transmission costs in (27) will make Alice select the maximum transmit power to transmit the signals.


## 5. Hierarchical Reinforcement Learning Framework for Secure Transmission Game

The proposed UAV secure communication problem with multiple AEs has been formulated as a Stackelberg game, which belongs to the category of two-stage dynamic game and has a significant two-layer game structure. Alice and all AEs become intelligent agents and have the learning ability to automatically optimize their configuration. Besides, the mixed-strategy is applied by both sides of the communication to confuse each other. In this section, we apply a hierarchical RL framework to derive the mixed-strategy equilibrium and implement the UAV secure communication.


### 5.1. Analysis of Mixed-Strategy Equilibrium

Considering the actual wireless communication scenario, we assume that Alice has a finite and discretized power set. Specifically, a policy of Alice at time slott is defined to be a probability vector πt=π1t,π2t,⋯πLt, where πlt means the probability with which Alice chooses action (power level) Pl from a finite discrete set P, which satisfies ∑l=1Lπlt=1. Similarly, δmt=δm,1mt,δm,2mt denotes the policy of AEm at time slot t, where δm,imt means the probability with which AEm chooses action (attack mode) Qi from a finite discrete set Q, which satisfies ∑i=12δm,imt=1.Based on the above analysis, we have the following definition of an SE for the hierarchical RL framework based on Eqs. (10) and (12). Alice’s objective is to maximize its revenue as
(29)π∗=argmaxπUaπ,δm∗,δ−m∗,Similarly, each AE’s objective is(30)δm∗=argmaxδmUmπ∗,δm,δ−m∗,Then, we will define the SE in a hierarchical reinforcement learning framework.Definition 1.
A stationary policy profileπ∗,δm∗,δ−m∗ is the SE for hierarchical RL framework if the followings hold.
(31)Uaπ∗,δm∗,δ−m∗≥Uaπ,δm∗,δ−m∗aUmπ∗,δm∗,δ−m∗≥Umπ∗,δm,δ−m∗b.Proposition 4.
For the proposed hierarchical RL framework, there exists Alice’s stationary policy and an AEs’ NE policy that form an SE.Proof.
If the Alice follows a stationary policyπ, the Stackelberg game is simplified into an M-player hierarchical RL game. It has been shown in [42] that every finite strategic-form game has a mixed policy equilibrium. As a result, there always exists an NE π in our formulation of the discrete power allocation game given Alice’s policy π. The rest of the proof follows directly from the definition of an SE and is thus omitted for brevity.


### 5.2. Hierarchical Q-Learning Based Power Allocation Algorithm

In the proposed UAV secure transmission game, since there is no information exchange between Alice and AEs, both sides can only maximize their expected utilities through repeated interactions with each other. When the action taken by the agent (Alice or AEs) brings positive feedback to the agent, the agent will strengthen the action, otherwise the agent will weaken the action. Agents constantly adjust their strategies based on the feedback to achieve optimal long-term returns. Thus, a hierarchical Q-learning based power allocation algorithm (HQLA) is adopted, where each agent’s policy is parameterized through the Q-function that characterizes the relative expected utility of a particular action.To be specific, for the follower’s learning, letQmtqm,imt denote the corresponding Q-function of AEm′s action qm,imt based on current police δm,imt at time slot t. Then, after conducting the action qm,imt, the corresponding Q-value is updated as follows
(32)Qmt+1qm,im=Qmtqm,imt+αUmPlt+1,qm,imt,q−m,i−mt+1−Qmtqm,imt,where α∈0,1 is the learning rate and UmPlt+1,qm,imt,q−m,i−mt+1 is the utility of AEm at time slot t+1.Each AE updates its policy based on Boltzmann distribution(33)δm,imt+1qm,im=expQmtqm,imt/τ∑qs,is∈QexpQmtqs,jst/τ,where temperature τ controls the trade-off between exploration and exploitation, i.e., for τ→0, AEm greedily chooses the policy corresponding to the maximum Q-value which means pure exploitation, whereas for τ→∞, AEm′s policy is completely random which means pure exploration [43]. Accordingly, the Q-value of Alice is updated as follows
(34)Qat+1Pl=QatPl+αUaPl,qm,imt+1,q−m,i−mt+1−QatPl,where UaPl,qm,imt+1,q−m,i−mt+1 is the utility of Alice at time slot t+1. Then, Alice updates its policy based on Boltzmann distribution
(35)πlt+1Pl=expQatPl/τ∑Pj∈PexpQatPj/τ.Now, we present the detailed description of the Q-learning based hierarchical RL algorithm.Algorithm 1.
Hierarchical Q-learning Based Power Allocation Algorithm
1: Initializet=0,QatPl=0,Qmtqm,imt=0,πltPl=1/L,δm,imt=1/2,m∈N\Alice;
2:Loop:
3:t=t+1;
4: Update Alice’s policiesπltPl and AEm′s policies δm,imtqm,im according to (35) and (33), respectively;
5: Alice chooses the actionPl with πlt;
6: EachAEm sensing the Alice’s transmit power, and selects qm,im with δm,imt;
7: Alice updatesUaPl,qm,imt+1,q−m,i−mt+1 according to (10), and each AEm updates UmPlt+1,qm,imt,q−m,i−mt+1 according to (12);
8: Alice and all AEs update Q-values according to (34) and (32), respectively;
9:End Loop;


### 5.3. Convergence Analysis of Algorithm1

The learning algorithm results in a stochastic process of choosing a power level, so we need to investigate the long-term behavior of the learning procedure. Along with the discussion in [43], we obtain the following differential equation describing the evolution of the Q-values:
(36)dQat+1Pldt=αUaPl,qm,imt+1,q−m,i−mt+1−QatPl,(37)dQmt+1qm,imdt=αUmPlt+1,qm,imt,q−m,i−mt+1−Qmtqm,imt.In the following, we would like to express the dynamics in terms of strategies rather than the Q-values. Toward this end, we differentiate (35) with respect to time t and use (36). Similarly, we differentiate (33) with respect to time t and use (37). We can obtain the equations like (38) and (39).
(38)dπlt+1Pldt=πlt+1PlατUaPl−∑j∈Pπjt+1PjUaPj−τ∑j∈Pπjt+1Pjlnπlt+1Plπjt+1Pj,(39)dδm,imt+1qm,imdt=δm,imt+1qm,imατUmqm,im−∑is∈Qδs,ist+1qs,isUmqs,is−τ∑is∈Qδs,ist+1qs,islnδm,imt+1qm,imδs,ist+1qs,is.The steady-state strategy profilezs=πsPl,δm,imsqm,im can be obtained [43].
(40)πsPl=expQatPl/τ∑Pj∈PexpQatPj/τ,(41)δm,imsqm,im=expQmtqm,imt/τ∑qs,is∈QexpQmtqs,jst/τ.LetZt=z1t,⋯,zNt the strategy profile of all players at time slot t. In the following analysis, we resort to an ordinary differential equation (ODE) whose solution approximates the convergence of Zt. The right-hand side of (38) and (39) can be represented by a function fZt as α→0. Zt will converge weakly to Z∗=π0∗,δ0∗, which is the solution to
(42)dZdt=fZ,Z0=Z0.Proposition 5.
The HQLA can discover a mixed-strategy SE.Proof.
We prove this by contradiction. Suppose that the process generated by (33) and (35) converges to a non-SE. But the solutions of (42) are by definition stationary points. This implies that HQLA will only converge to stationary points. This means that stationary points that are not SEs are stable, which is contradicting Proposition 4.


## 5.1. Analysis of Mixed-Strategy Equilibrium

Considering the actual wireless communication scenario, we assume that Alice has a finite and discretized power set. Specifically, a policy of Alice at time slott is defined to be a probability vector πt=π1t,π2t,⋯πLt, where πlt means the probability with which Alice chooses action (power level) Pl from a finite discrete set P, which satisfies ∑l=1Lπlt=1. Similarly, δmt=δm,1mt,δm,2mt denotes the policy of AEm at time slot t, where δm,imt means the probability with which AEm chooses action (attack mode) Qi from a finite discrete set Q, which satisfies ∑i=12δm,imt=1.Based on the above analysis, we have the following definition of an SE for the hierarchical RL framework based on Eqs. (10) and (12). Alice’s objective is to maximize its revenue as
(29)π∗=argmaxπUaπ,δm∗,δ−m∗,Similarly, each AE’s objective is(30)δm∗=argmaxδmUmπ∗,δm,δ−m∗,Then, we will define the SE in a hierarchical reinforcement learning framework.Definition 1.
A stationary policy profileπ∗,δm∗,δ−m∗ is the SE for hierarchical RL framework if the followings hold.
(31)Uaπ∗,δm∗,δ−m∗≥Uaπ,δm∗,δ−m∗aUmπ∗,δm∗,δ−m∗≥Umπ∗,δm,δ−m∗b.Proposition 4.
For the proposed hierarchical RL framework, there exists Alice’s stationary policy and an AEs’ NE policy that form an SE.Proof.
If the Alice follows a stationary policyπ, the Stackelberg game is simplified into an M-player hierarchical RL game. It has been shown in [42] that every finite strategic-form game has a mixed policy equilibrium. As a result, there always exists an NE π in our formulation of the discrete power allocation game given Alice’s policy π. The rest of the proof follows directly from the definition of an SE and is thus omitted for brevity.


## 5.2. Hierarchical Q-Learning Based Power Allocation Algorithm

In the proposed UAV secure transmission game, since there is no information exchange between Alice and AEs, both sides can only maximize their expected utilities through repeated interactions with each other. When the action taken by the agent (Alice or AEs) brings positive feedback to the agent, the agent will strengthen the action, otherwise the agent will weaken the action. Agents constantly adjust their strategies based on the feedback to achieve optimal long-term returns. Thus, a hierarchical Q-learning based power allocation algorithm (HQLA) is adopted, where each agent’s policy is parameterized through the Q-function that characterizes the relative expected utility of a particular action.To be specific, for the follower’s learning, letQmtqm,imt denote the corresponding Q-function of AEm′s action qm,imt based on current police δm,imt at time slot t. Then, after conducting the action qm,imt, the corresponding Q-value is updated as follows
(32)Qmt+1qm,im=Qmtqm,imt+αUmPlt+1,qm,imt,q−m,i−mt+1−Qmtqm,imt,where α∈0,1 is the learning rate and UmPlt+1,qm,imt,q−m,i−mt+1 is the utility of AEm at time slot t+1.Each AE updates its policy based on Boltzmann distribution(33)δm,imt+1qm,im=expQmtqm,imt/τ∑qs,is∈QexpQmtqs,jst/τ,where temperature τ controls the trade-off between exploration and exploitation, i.e., for τ→0, AEm greedily chooses the policy corresponding to the maximum Q-value which means pure exploitation, whereas for τ→∞, AEm′s policy is completely random which means pure exploration [43]. Accordingly, the Q-value of Alice is updated as follows
(34)Qat+1Pl=QatPl+αUaPl,qm,imt+1,q−m,i−mt+1−QatPl,where UaPl,qm,imt+1,q−m,i−mt+1 is the utility of Alice at time slot t+1. Then, Alice updates its policy based on Boltzmann distribution
(35)πlt+1Pl=expQatPl/τ∑Pj∈PexpQatPj/τ.Now, we present the detailed description of the Q-learning based hierarchical RL algorithm.Algorithm 1.
Hierarchical Q-learning Based Power Allocation Algorithm
1: Initializet=0,QatPl=0,Qmtqm,imt=0,πltPl=1/L,δm,imt=1/2,m∈N\Alice;
2:Loop:
3:t=t+1;
4: Update Alice’s policiesπltPl and AEm′s policies δm,imtqm,im according to (35) and (33), respectively;
5: Alice chooses the actionPl with πlt;
6: EachAEm sensing the Alice’s transmit power, and selects qm,im with δm,imt;
7: Alice updatesUaPl,qm,imt+1,q−m,i−mt+1 according to (10), and each AEm updates UmPlt+1,qm,imt,q−m,i−mt+1 according to (12);
8: Alice and all AEs update Q-values according to (34) and (32), respectively;
9:End Loop;


## 5.3. Convergence Analysis of Algorithm1

The learning algorithm results in a stochastic process of choosing a power level, so we need to investigate the long-term behavior of the learning procedure. Along with the discussion in [43], we obtain the following differential equation describing the evolution of the Q-values:
(36)dQat+1Pldt=αUaPl,qm,imt+1,q−m,i−mt+1−QatPl,(37)dQmt+1qm,imdt=αUmPlt+1,qm,imt,q−m,i−mt+1−Qmtqm,imt.In the following, we would like to express the dynamics in terms of strategies rather than the Q-values. Toward this end, we differentiate (35) with respect to time t and use (36). Similarly, we differentiate (33) with respect to time t and use (37). We can obtain the equations like (38) and (39).
(38)dπlt+1Pldt=πlt+1PlατUaPl−∑j∈Pπjt+1PjUaPj−τ∑j∈Pπjt+1Pjlnπlt+1Plπjt+1Pj,(39)dδm,imt+1qm,imdt=δm,imt+1qm,imατUmqm,im−∑is∈Qδs,ist+1qs,isUmqs,is−τ∑is∈Qδs,ist+1qs,islnδm,imt+1qm,imδs,ist+1qs,is.The steady-state strategy profilezs=πsPl,δm,imsqm,im can be obtained [43].
(40)πsPl=expQatPl/τ∑Pj∈PexpQatPj/τ,(41)δm,imsqm,im=expQmtqm,imt/τ∑qs,is∈QexpQmtqs,jst/τ.LetZt=z1t,⋯,zNt the strategy profile of all players at time slot t. In the following analysis, we resort to an ordinary differential equation (ODE) whose solution approximates the convergence of Zt. The right-hand side of (38) and (39) can be represented by a function fZt as α→0. Zt will converge weakly to Z∗=π0∗,δ0∗, which is the solution to
(42)dZdt=fZ,Z0=Z0.Proposition 5.
The HQLA can discover a mixed-strategy SE.Proof.
We prove this by contradiction. Suppose that the process generated by (33) and (35) converges to a non-SE. But the solutions of (42) are by definition stationary points. This implies that HQLA will only converge to stationary points. This means that stationary points that are not SEs are stable, which is contradicting Proposition 4.


## 6. Simulation Results

Simulations are carried out to evaluate the performance of the proposed power allocation strategies against multiple UAV AEs. This scenario has one transmitter-receiver pair and three UAV AEs denoted as Alice, Bob,AE1, AE2, and AE3, respectively. We set up a scenario network where all the UAVs are distributed in a 200 m ∗ 200 m region. The system parameters are chosen for some typical scenarios including the cost of unit transmit power and jamming power, i.e., Ca=Cj=0.1 and Ce=0.5, the path loss exponent η=2 and ω0=80.Figure2 shows the expected utilities of the leader under different algorithms. We can find that the expected utility achieved by the proposed HQLA is significantly lower than the single-agent Q-learning algorithm (SAQL). This is because in SAQL, only Alice applies the reinforcement learning mechanism to maximize the secrecy rate but all AEs’ behaviors constituting joint actions are considered to be stated in the Q-learning algorithm which means each AE cannot choose the optimal strategy adaptively to maximize its utility. While in HQLA, each AE with reinforcement learning ability can maximize its damages to the secrecy rate of the considered system through repeated interactions with Alice’s and other AEs’ strategies. The comparison of the leader’s expected utilities with SAQL implies that the agent’s learning ability has a significant impact on its utility. So, the proposed HQLA provides an optimal power allocation strategy in a more hostile case that suffers the adaptive attacks from multiple AEs. On the other hand, the proposed HQLA is superior to the random selection algorithm (RS) because the proposed HQLA may converge to a desirable solution, whereas the RS is an instinctive approach.Figure 2
Expected utility of leader.Figure3 shows the cumulative distribution function (CDF) of the convergence of HQLA and SAQL. As observed from Figure 3, we can find that the proposed algorithm converges at about 500 iterations, while the contrast algorithm converges at about 1000 iterations, which means the convergence rate of HQLA is significantly better than SAQL. This is because that all AEs in SAQL select action randomly without learning ability whereas in HQLA, taking the interactions between two sides of communication into account, all AEs make decisions according to the mixed-strategy derived by RL which can obtain the optimal strategy via trials-and-errors. This also means that the learning ability has a significant positive impact on the convergence rate.Figure 3
The CDF of convergence of HQLA and SAQL.Figure4 presents the strategy selection probabilities evolution of the leader’s transmit power. At the very beginning, Alice randomly selects transmit power according to a uniform distribution. As Algorithm 1 iterates, the strategy selection probabilities keep on updating until convergence after about 500 iterations. It is worthy to note that Algorithm 1 under this scenario converges to pure strategy NE points since the probability of selecting one power level is equal to 1, while the probabilities of the other levels of transmit power decrease to 0 if the time slots are large enough. So, the theoretical prediction in Proposition 4 is verified under the existing conditions. Specifically, the Pmax in Figure 4(a) as the optimal transmit power is consistent with Proposition 3, and the Ps∗ in Figure 4(b) is consistent with Proposition 2.Figure 4
Power allocation probabilities of leader.
(a)
(b)The leader’s expected utility comparison under differentCe is shown in Figure 5(a). It is noted that the steady value of the leader’s expected utility increase with the value of Ce growing because that Ce leads to changes in AEs’ attack strategies. Specifically, as a rational agent, all AEs choose to interfere with Bob finally in Figure 5(b) because they find the utility of the jammer is higher than eavesdropper according to the learning process when the difference of Ce=0.8. Thus, the maximal data rate of the Alice-AE link is zero which means the leader will obtain the maximal secrecy rate and expected utility. Similarly, in Figure 5(d), all AEs find the utility of eavesdropper is higher than jammer when Ce=0.2 and every AE choose to eavesdrop on Alice. As a result, the maximal data rate of the Alice-AE link between all AEs is achieved and the leader suffers the lowest utility. When Ce=0.5 (in Figure 5(c)), according to the utilities of themselves, AE1 and AE3 always choose to interfere with Bob and AE2 prefers eavesdropping which makes the expected utility of leader is between Ce=0.8 and Ce=0.2. In addition, it is worthy to note that the attack strategies of all AEs have a pure strategy equilibrium since the probability of selecting one attack mode is equal to 1 while the probability of another attack mode decreases to 0.Figure 5
Expected utility of leader and attack mode probabilities of all AEs under differentCe.
(a)
Expected utility of leader under differentCe
(b)
C
e
=
0.8
(c)
C
e
=
0.5
(d)
C
e
=
0.2


## 7. Conclusions and Future Work

In this paper, we have investigated the transmit power optimization problem of secure UAV communication in the presence of multiple UAV AEs. A secure transmission game is formulated to prove the existence of the NE by analyzing the interactions between the legitimate user and AEs. Within a hierarchical game framework, we obtain the optimal transmit power solutions for the legitimate transmissions. Numerical results verified the theoretical analysis and shown that the secrecy performance could be degraded severely by AEs’ learning ability. Moreover, the outperformance of the HQLA’s convergence and the impact of the eavesdropping cost on the decision of AE’s attack mode is also demonstrated. To take advantage of the UAV’s mobility that can bring the potential performance enhancement, in future work, we will devote our efforts to joining the UAV’s trajectory and resource allocation optimization against multiple AEs.



---

*Source: 8825120-2020-10-08.xml*