## 3 The Joint Relay Selection and Power Control Optimization Method via Stackelberg Game
### Hierarchical Joint Optimization Algorithm  
In this section, we propose a hierarchical joint optimization algorithm, which realizes the user's strategy optimization based on Q-learning algorithm [47] and jammer's strategy optimization based on multi armed bandit (MAB) method [48; 49; 50], respectively. For the user, its mixed power and relay selection strategy at the \(t\)th time slot is expressed as \(\mathrm{q}(t)=\{q_{1,1}(t),q_{1,2}(t),...,q_{1,N}(t),...,q_{m,n}(t),...,q_{M,1 }(t),q_{M,2}(t),...,q_{M,N}(t)\}\), where \(q_{m,n}(t)\) denotes the probability to select the \(n\)th relay and power \(P_{m}\) at the \(t\)th time slot. Then we update the user's Q value as follows:  
\[Q_{m,n}(t+1)=(1-\kappa^{t})Q_{m,n}(t)+\kappa^{t}r(t), \tag{14}\]  
where \(\kappa^{t}\) denotes the user's learning rate at the \(t\)th time slot, which satisfies \(\sum\limits_{t=0}^{\infty}\kappa^{t}=\infty,\sum\limits_{t=0}^{\infty}(\kappa ^{t})^{2}<\infty\). \(r(t)\) denotes the user's reward, i.e., its utility \(U\) at the \(t\)th time slot. Based on the analysis above, the user's mixed strategy is updated as follows:  
\[q_{m,n}(t+1)=\frac{\exp\left[q_{m,n}(t)\big{/}\tau_{0}\right]}{ \sum\limits_{1\leq m\leq M,1\leq n\leq N}\exp\left[q_{m,n}(t)\big{/}\tau_{0} \right]}, \tag{15}\]  
where \(\tau_{0}\) controls the tradeoff of exploration-exploitation. For the jammer, we formulate the finite strategy optimization as a MAB problem, and consider each power strategy as a arm to select, i.e., the jamming power strategy \(J_{l}\) is considered as the \(l\)th arm. \(J(k)\) denotes the jammer's power strategy at the \(k\)th epoch. Thus the times of power \(J_{l}\) has been selected in the past \(K\) epochs is defined as:  
\[A_{l}(K)=\sum\limits_{k=1}^{K}\delta(J_{l},J(k)), \tag{16}\]  
where \(\delta(x,y)\) is the Kronecker delta function and it can be expressed as:  
\[\delta(x,y)=\left\{ \begin{array}{ll}1&x=y\\ 0&x\
eq y\end{array} \right. \tag{17}\]  
Thus, the jammer's \(l\)th arm's statistical average reward \(\mu_{l}\) in the past \(K\) epochs is defined as:  
\[\mu_{l}=\frac{\sum\limits_{k=1}^{K}V(k)\delta(J_{l},J(k))}{A_{l}(K)}. \tag{18}\]  
\(B(K)\) denotes the jammer's total utility in the past \(K\) epochs and is expressed as:  
\[B(K)=\sum\limits_{k=1}^{K}V(k). \tag{19}\]Inspired by [48; 49; 50], we define jammer's regret in the past \(K\) epochs, which is an important index to represent the loss of utility so far because of the failure to select the optimal jamming power strategy, as shown in the following:  
\[E\left[R(K)\right] =E\left[B^{*}(K)\right]-E\left[B(K)\right]\] \[=\sum\limits_{l\
eq l^{*}}\left(V_{l^{*}}-V_{l}\right)E\left[A_{l} (K)\right], \tag{20}\]  
where \(l^{*}\) denotes the estimated optimal arm of jammer based on the historical statistical average reward. \(B^{*}(K)\) denotes jammer's accumulative utility it could get if jammer had always chosen \(l^{*}\) in the past \(K\) epochs. In the MAB problem, UCB1 [48; 49; 50] is an effective policy to realize the optimization of power strategy. Adopting the UCB1 method, jammer updates its strategy in the \((K+1)\)th epoch based on the following condition:  
\[J(K+1)=\underset{l\in[1,2\ldots,L]}{\arg\max}\left(\mu_{l}+\sqrt{\frac{2\ln K }{A_{l}(K)}}\right). \tag{21}\]  
According to the analysis above, based on the Q-learning algorithm and UCB1 method, both the user's and jammer's optimal strategy can be obtained through hierarchical joint optimization algorithm, which is shown in Algorithm 1. ```
Initialization:  The number \(T\) of time slots in one epoch, the number \(k_{\max}\) of all epochs. User selects power \(P\) randomly, \(t=1\), \(k=0\). Outer Iteration:  1. Jammer selects the optimal arm \(J(k+1)\) according to Equation (21). Inner Iteration:  (1) In the \(t\)th time slot, the user selects its joint relay and power strategy according to \(\mathfrak{q}(t)\). (2) Obtain the user's utility \(U(t)\) through the feedback of SINR at the end of current time slot  according to Equation (5). (3) Update the user's Q values according to Equation (14).