# An Intelligent Access Channel Algorithm Based on Distributed Double Q Learning
## 3 Anti-Jamming Algorithm of Intelligent Communication Based on Distributed Double Q-Learning
According to the literature [25], Q value can be updated as follows:  
\[Q_{1,t+1}(n,k)=(1-a)Q_{1,t}(n,k)+\alpha*R_{1}(n,k) \tag{18}\]  
\[Q_{2,t+1}(n,m)=(1-\alpha)Q_{2,t}(n,m)+\alpha*R_{2}(n,m) \tag{19}\]  
where, \(\alpha\) is the learning rate, and our proposed algorithm takes actions that only consider the current reward and do not consider the future reward. The discount factor of the future reward \(\gamma\) is 0, because once the user successfully communicates, their needs are satisfied  
The base station can obtain the channel collision status of users in each time slot through preamble detection [26]. The subframe occupancy intensity and the channel collision state can be broadcast to all users through the base station, reducing the interaction between users, signaling overhead and user edge computation complexity. The simplest way to implement the Q-learning algorithm is to use the \(\varepsilon\)-greedy method of trade-off ex ploration; that is, each user chooses the action with the highest Q value with the probability of \(1-\varepsilon\), and randomly chooses an action in the action space with the probability of \(\varepsilon\). The optimal policy of this user is to execute his actions according to the \(\varepsilon\)-greedy policy. The greedy policy selects the actions with the highest Q value in subframe-Q and channel-Q tables with a probability of \(1-\varepsilon\), namely:  
\[\pi_{n}^{*}(k,m)=\left\{ \begin{array}{l}\underset{k,m}{\operatorname{argmax} }\{Q_{1}(n,k),Q_{2}(n,m)\}\text{ Probability of }1-\varepsilon\\ \forall_{k},\forall_{m}\text{ Probability of }\varepsilon\end{array} \right. \tag{20}\]  
For the user \(n\), the specific process of the proposed algorithm is shown in Algorithm 1. In the first iteration, the values of the two Q tables are all initialized to 0, and each user randomly selects any subframe and channel. After receiving the feedback signal from the base station, Equations (10) and (11) are used to calculate the reward \(R_{1}(n,k)\) and \(R_{2}(n,m)\) brought by the action selected by the user, and then Equations (18) and (19) are used to update the corresponding Q values in the time slot and channel table. After each iteration of the Q table, the user selects the channel corresponding to the maximum Q value in the channel-q table, and then selects the subframe corresponding to the maximum Q value in the subframe -Q table to send the packet. When the number of iterations does not reach the maximum number of iterations or the convergence result is not reached, the DDQL algorithm will continue to update iteratively. The processing flow of user \(n\) DDQL algorithm is shown in Algorithm 1. We first initialize the parameters, the learning factor \(\alpha\), the maximum number of iterations T, and then set all Q values in the subframe-Q table and channel-Q table of each user to 0. In the first iteration (\(t=1\)), each user selects a channel in a random manner and sends a packet on a subframe selected in a random manner. Later, when each user feedback signal received from the base station, through the Equations (18) and (19), update the subframe Q table, respectively, and channel--Q value in the table, where each iteration after the update, the user choosing the Q value of the highest channel--Q value and the corresponding channel, and select in the table in a subframe--Q value of the highest in the corresponding subframe to transmit packets. Until the number of iterations T reaches the maximum number of iterations L or the system converges, the DDQL algorithm will continue to be executed until each user can obtain the best subframes and channels, thus obtaining the optimal solution to problem (1). ```
1:Initialization: \(Q_{1,t}(n,k)=0\), \(Q_{2,t}(n,m)=0\), \(t=1\), \(\forall k\in[1,\ldots,K]\), \(\forall m\in[1,\ldots,M]\). Set the learning factor \(\alpha\) and the maximum number of iterations \(T\).