## 3 RL-Based Jamming Algorithm
### Q-Learning  
Q-learning is one of the most popular RL algorithms. The core updated formula [35] is given by:  
\[Q(u_{t-1},a_{t-1})\leftarrow(1-\alpha)Q(u_{t-1},a_{t-1})+\alpha[R+\gamma\max_{a }Q(u_{t},a)], \tag{6}\]  
where \(Q(u_{t-1},a_{t})\) is the Q-value that evaluates the action \(a_{t}\) at state \(u_{t-1}\)(we set \(\tau=1\)), \(\alpha\in(0,1)\) is the learning rate, \(R\) is the reward value.  
In actual situation, a jammer could select \(n\) frequency bands to jam in decision set \(\mathcal{A}_{j}\) (under guidance of agent). At time \(t\), the decision is selected from \(Q(u_{t,c},\mathcal{A}_{j})\) (\(Q\) is a two-dimensional matrix), note \(\mathcal{A}_{s,t}\) is the top \(n\) values selected from \(Q(u_{t,c},\mathcal{A}_{j})\) at time \(t\), define this operation as \(sort[Q(u_{t,c},\cdot)]_{1:n}\). \(R_{s,t}\) is the reward vector for corresponding actions. For example, if the actions \(\mathcal{A}_{s,t}=\{a_{1,t},a_{2,t},a_{4,t}\}\) and if \(a_{4,t}\), \(R=1\),then \(R_{s,t}=\{0,0,0,1,0,...,0\}\), \(\big{(}\big{|}\mathcal{A}_{j}\big{|}=|R_{s,t}|=M\big{)}\). The update formula of jammer is as follows:  
\[Q(u_{t-1,c},\mathcal{A}_{j})\leftarrow(1-\alpha)Q(u_{t-1,c},\mathcal{A}_{j})+ \alpha[R_{s,t}+\gamma Q(u_{t,c},\mathcal{A}_{j})]. \tag{7}\]  
The reinforcement learning jamming algorithm is repeated until any stop criterion is met (see Algorithm 1).  
```
Initialize: Set learning rate \(\alpha\in(0,1)\), discount rate \(\gamma\in(0,1)\), \(Q=0\), training time \(T\), and probability of random action \(\varepsilon\in(0,1)\). The number of jamming frequency bands is \(n\). for\(t=1,2,...,\infty\)do  Generate random value \(\eta\in(0,1)\) if\(t<T\)then  Agent chooses \(n\) action from \(\mathcal{A}_{j}\) randomly (\(a_{t}\in\mathcal{A}_{j},\text{P}(a_{t}=a)=\frac{1}{N}\), \(N=\big{|}\mathcal{A}_{j}\big{|}\)) gets reward,  update Q matrix: \(Q(u_{t-1,c},\mathcal{A}_{j})\leftarrow(1-\alpha)Q(u_{t-1,c},\mathcal{A}_{j})+ R_{s,t}+\alpha[R+\gamma Q(u_{t,c},\mathcal{A}_{j})]\) if\(\eta>\varepsilon\)then  Agent choose action top \(n\) action, \(sort[Q(u_{t,c},\cdot)]_{1:n}\), \(\mathcal{A}_{s,t}\), gets reward, updates Q matrix: \(Q(u_{t-1,c},\mathcal{A}_{j})\leftarrow(1-\alpha)Q(u_{t-1,c},\mathcal{A}_{j})+ R_{s,t}+\alpha[R+\gamma Q(u_{t,c},\mathcal{A}_{j})]\) else  Agent chooses \(n\) action from \(\mathcal{A}_{j}\) randomly, gets reward, updates Q matrix: \(Q(u_{t-1,c},\mathcal{A}_{j})\leftarrow(1-\alpha)Q(u_{t-1,c},\mathcal{A}_{j})+ R_{s,t}+\alpha[R+\gamma Q(u_{t,c},\mathcal{A}_{j})]\) endif else  Finish training endif  Agent guides actions based on Q matrix endfor
```  
**Algorithm 1** Reinforcement learning jamming algorithm (RLJA).  
Theorem 1: _The jamming process of RL algorithm can converge with different \(\varepsilon\) under condition that the communication process is action replay process (ARP)._  
Proof: The proof is given in reference [35], which is divided into two parts. The first part explains what action-replay process (ARP) is. ARP is derived from a particular sequence of episodes observed in the real process, with a finite state space \(\mathcal{S}\) and a finite action set \(\mathcal{A}\). Second part proves that Q-learning algorithm can converge under condition that the process in real situation is ARP. Exploring conditionsare not requirements to the proof, the iterations are. Since the jamming pattern without a random scheme in this case is ARP, iterations are enough. Therefore, the algorithm can converge.