# An Intelligent Access Channel Algorithm Based on Distributed Double Q Learning
## 2 System Model and Problem Formulation
### Problem Formulation
\((k_{j},m_{j})\) denotes the subframe and channel where the jamming is located. \((k_{i},m_{i})\) denotes the subframe and channel selected by other users except user \(n\). Furthermore, we model the channel and subframe selection process as two Markov decision processes (MDP), and equivalently prove that the two-element joint selection optimization problem of the subframe channel is equivalent to two unitary optimization problems of the subframe channel, and each access subframe and channel are, respectively, treated as the states of different MDP. In the Markov decision process, each agent constantly interacts with the environment, queries the Q table according to the current state of the environment, selects the optimal action, so as to obtain the return brought by the action, and then enters the next state. The current states of the two Markov processes defining access subframe and channel are, respectively, each access subframe and channel, so the two state spaces are:  
\[\mathbf{S_{k}}=\{1,2,\ldots,K\} \tag{5}\]  
\[\mathbf{S_{m}}=\{1,2,\ldots,M\} \tag{6}\]  
The current action spaces of the two Markov processes defining access subframe and channel are, respectively, each access subframe and channel, so the two action spaces are:  
\[\mathbf{A_{k}}=\{1,2,\ldots,K\} \tag{7}\]  
\[\mathbf{A_{m}}=\{1,2,\ldots,M\} \tag{8}\]In the existing literature, the reward function \(R_{1}(n,k)\) of user \(n\) in the \(kth\) time slot is called binary classical Q-learning, and its expression is as follows:  
\[R_{1}(n,k)=\left\{ \begin{array}{ll}+1&\text{if transmission succeeds}\\ -1&\text{otherwise}\end{array} \right. \tag{9}\]  
We define the occupancy intensity of a subframe as the number of users that a user intends to transmit packets in the same subframe; that is \(\mathbf{Z}\mathbf{L}=[ZL_{1},ZL_{2},\dots,ZL_{K}]\), where \(\mathbf{Z}\mathbf{L}\) is a one-dimensional vector of \(1\times K\). On the basis of occupancy intensity, we define the Markov process reward function for accessing subframes as \(R_{1}(n,k)\):  
\[R_{1}(n,k)= \begin{cases}+1&\text{when }\mathbf{Z}\mathbf{L}(k)\leq(M-\text{Z}) \\ -\mathbf{Z}\mathbf{L}(k)/K&\text{when }\mathbf{Z}\mathbf{L}(k)>(M-Z)\end{cases}  \tag{10}\]  
where, \(R_{1}(n,k)\) is the reward function for the user \(n\) to select a subframe, \(n\in\{1,\dots,N\}\), \(k\in\{1,\dots,K\}\), \(m\in\{1,\dots,M\}\). The number of channels jammed by the jammer is \(\mathbf{Z}\mathbf{L}(k)\) represents the user occupancy intensity of the \(kth\) subframe. That is, the number of users who intend to transmit packets in the same subframe. If the user occupancy intensity in the \(kth\) subframe is less than or equal to the number of all channels \(M-Z\), then \(R_{1}(n,k)\) equals 1, indicating that the transmission is successful; Otherwise, it equals \(-\mathbf{Z}\mathbf{L}(k)/K\), indicating that the transmission fails. The Markov process reward function of access channel is defined as \(R_{2}(n,m)\):  
\[R_{2}(n,m)=\left\{ \begin{array}{ll}+1&\text{when }n\in S(n^{\prime})\\ -1&\text{when }n\
otin S(n^{\prime})\end{array} \right. \tag{11}\]  
where, \(R_{2}(n,m)\) is the reward function for the user \(n\) to select a channel, \(k\in\{1,\dots,K\}\),\(m\in\{1,\dots,M\}\). The set of users who occupy idle channel collection is \(S(n^{\prime})\). \(n\in S(n^{\prime})\) represents user \(n\) select the \(mth\) channel that is idle channel in the \(kth\) subframe. There is no other user interference and jamming, otherwise the user \(n\) in the \(kth\) subframe select the \(mth\) channel where other user or jammer occupying. It can lead to channel conflict or disturbance of communication signals, lead to the failure signals. If the \(mth\) channel selected by user \(n\) in the \(kth\) subframe is an idle channel, then \(R_{2}(n,m)\) is equal to 1, indicating that the transmission is successful; Otherwise, it is equal to \(-1\), indicating that the transmission fails.