# An Intelligent Access Channel Algorithm Based on Distributed Double Q Learning
## 3 Anti-Jamming Algorithm of Intelligent Communication Based on Distributed Double Q-Learning  
To solve the problem shown in Equation (1), we propose an anti-jamming algorithm based on distributed double Q-learning. Each user uses two similar Q-learning algorithms to select channel and sub-frame, respectively, according to the occupancy strength of channel and subframe. In the Markov decision process, each agent constantly interacts with the environment, queries the Q table according to the current state of the environment, selects the optimal action, so as to obtain the return brought by the action, and then enters the next state. As mentioned earlier, the goal of the proposed algorithm is to maximize the sum of the throughput of all users in the wireless communication network. First, we prove that two unary Q learning algorithms and a binary Q learning algorithm (subframe is row, channel is column) are equivalent; that is, we prove whether the binary optimization problem of the joint selection of subframe and channel is equivalent to two unary optimization problems of subframe selection and channel selection. According to the optimization theory [24], the necessary and sufficient condition for equivalence is that the two independent variables are separable. Prove that \(Q_{2}(n,m)=Q_{n}(k_{n},m_{n})\), \(Q_{1}(n,k)\) is the Q value of the selected subframe, \(Q_{2}(n,m)\) is the Q value of selected channel. \(Q_{n}(k_{n},m_{n})\) is the Q value of selected subframe and channel together. In Equation (4), \((k_{n},m_{n})\
eq\big{(}k_{j},m_{j}\big{)}\,\&\,(k_{n},m_{n})\
eq(k_{i},m_{i})(i \in N/n)\) indicates that the user \(n\) does not suffer from inter-user interference or external malicious jamming. It fits the characteristics of the element in \(S(n^{\prime})\). The \(mth\) channel selected by user \(n\) in the \(kth\) subframe is the idle channel. Therefore, Equation (9) can be expressed as:  
\[R_{n}(k_{n},m_{n})= \begin{cases}1&n\in S(n^{\prime})\\ -1&\text{otherwise}\end{cases}  \tag{12}\]  
\(R_{1}(n,k)R_{2}(n,m)\) is:  
\[R_{1}(n,k)R_{2}(n,m)= \begin{cases}+1&\text{when }n\in S(n^{\prime})\\ -1&\text{otherwise}\end{cases}  \tag{13}\]  
According to Equations (12) and (13):  
\[R_{1}(n,k)R_{2}(n,m)=R_{n}(k_{n},m_{n}) \tag{14}\]  
According to the literature [14], the relationship between Q value and return function is as follows:  
\[Q(s,a)=r\big{(}s^{\prime}|s,a\big{)}+\gamma\sum_{s^{\prime}\in S}P^{a}_{ss^{ \prime}}\underset{a\in A}{\text{max}}Q(s^{\prime},a^{\prime}) \tag{15}\]  
where, \(s^{\prime}\) is the state of the next subframe, and \(P^{a}_{ss^{\prime}}\) is the state transition probability that is transferred from state \(s\) to state \(s^{\prime}\) after action \(a\) is taken by state \(s\). \(r(s^{\prime}|s,a)\) represents the immediate reward transferred to state \(s^{\prime}\) after action \(a\) is taken by state \(s\). The action taken in this paper only considers the current reward and does not consider the future reward. That is, the discount factor \(\gamma\) of the future reward is 0, so Equation (15) can be expressed as:  
\[Q(s,a)=r\big{(}s^{\prime}|s,a\big{)} \tag{16}\]  
According to Equations (14) and (16):  
\[Q_{1}(n,k)Q_{2}(n,m)=Q_{n}(k_{n},m_{n}) \tag{17}\]  
As can be seen from Equation (17), \(k\) and \(m\) are separable, so the algorithm proposed in this paper is equivalent to the subframe and channel joint algorithm. Therefore, we transform a two-element optimization problem of joint selection of subframe channels into two unary optimization problems of the separate selection of the subframe and channel. Each user needs to maintain two one-dimensional Q tables, namely user-subframe Q table and user-channel Q table. We denote the \(t-th\) iteration Q value of user \(n\) transmitting a packet in the \(kth\) subframe as \(Q_{1,t}(n,k)\), and the \(t-th\) iteration Q value of user \(n\) transmitting a packet in the \(mth\) channel as \(Q_{2,t}(n,m)\).