## 4 Hypothesis Test and Detection Method
### Impersonation Detection Method  
The Q-learning algorithm is an off-policy algorithm, whereas the SARSA algorithm is an on-policy algorithm [27]. Both of them are reinforcement learning algorithms that optimize the IAD strategy to a certain extent in a dynamic environment. Using continuous learning, the proposed method selects an optimal test threshold to detect impersonation attacks. According to the SARSA algorithm, the receiving ends choose specific suboptimal actions on each state with the probability of \(\varepsilon\) based on the \(\varepsilon\)-greedy strategy. In the \(\varepsilon\)-greedy strategy, the receiver perceives the state in each time slot, which is denoted by \(L\), and detects the legitimate and illegitimate nodes in each state. Upon perceiving the state, it performs action randomly with a probability of \(\varepsilon\) and decides the optimal action with a \(1-\varepsilon\). In short, the probability of choosing a more significant action value is \(1-\varepsilon\). Hence, the probability is given as  
\[P(\lambda)=\left\{\begin{matrix}1-\varepsilon,\lambda=\lambda^{*},\\ \frac{\varepsilon}{L},\lambda\in\{l/L\},\lambda\leq l\leq L,\lambda\
eq\lambda ^{*},\end{matrix}\right. \tag{8}\]  
where \(\lambda\) represents the test threshold value and is selected from \(L+1\) states, i.e., \(\lambda\in\{l/L\}\). In each state, regardless of which action is chosen, the immediate reward can be obtained as  
\[R=\sum_{t=(\tau-1)T+1}^{\tau T}\ U_{N}^{t}(\lambda,\kappa), \tag{9}\]  
where \(T\) shows the number of transmitted signals in a time slot, \(U_{N}^{t}\) is the immediate utility function, and \(\kappa\) is a set of training signals transmitted by illegitimate nodes. Similar to [27], the Bayesian risk of impersonation detection, \(I(\lambda,\kappa)\), can be expressed as  
\[I(\lambda,\kappa)=(g_{0}(1-P_{1}(\lambda))-c_{0}P_{1}(\lambda))(1-\sum_{j=1}^ {F}\ p_{i})+ \tag{10}\]  
The first part of Equation (10) represents the gain of legal users, while the second part represents the gain of illegitimate users. Here, \(g_{0},g_{1}\) represent the gain of accepting the legal packet and the gain of rejecting the illegal packet, respectively. Likewise, \(c_{0},c_{1}\) show the cost of rejecting the legal packet and the cost of receiving the illegal packet, respectively. If \(R_{0}\) is the immediate utility gained by accepting the legal packet and \(R_{1}\) is the primary utility gained by rejecting the illegal packet, then Equation (10) can be rewritten as  
\[R_{0}(\lambda,\kappa)=(g_{1}-g_{0}) \sum_{i=1}^{F}p_{i}-(g_{1}+g_{0})P_{2}(\lambda)\sum_{i=1}^{F}p_{i}- \tag{11}\] \[(g_{1}+c_{0})P_{1}(\lambda)(1-\sum_{i=1}^{F}\ p_{i})+g_{0},\]  
and  
\[R_{1}(\lambda,\kappa)=(g_{0}-g_{1}\ ) \sum_{i=1}^{F}p_{i}+(g_{1}+g_{0})P_{2}(\lambda)\sum_{i=1}^{F}p_{i}+ \tag{12}\] \[(g_{1}+c_{0})P_{1}(\lambda)(1-\sum_{i=1}^{F}\ p_{i})-g_{0}\]  
In the SARSA algorithm, the discount rate \(\eta,\eta\in(0,1)\) determines the importance of future rewards, whereas the learning rate, \(\alpha,\alpha\in(0,1)\), means the retention of previous learning outcomes. Hence, the updated value at the receiver side can be obtained as follows.  
\[Q(s_{t},\lambda)=(1-\alpha)Q(s_{t},\lambda)+\alpha\big{(}R+\eta Q\big{(}s_{t +1},\lambda^{\prime}\big{)}\big{)} \tag{13}\]  
and  
\[V(s_{t})=\text{m}\ Q(s_{t},\lambda),\lambda\in\{l/L\},\lambda\leq l\leq L \tag{14}\]  
The optimal test threshold is:  
\[\lambda^{*}=\text{arg}\ \text{m}\ Q(s_{t},\lambda),\lambda\in\{l/L\},\lambda \leq l\leq L \tag{15}\]  
The SARSA algorithm with the IAD method is detailed in Algorithm 1.