# An Intelligent Access Channel Algorithm Based on Distributed Double Q Learning
## 2 System Model and Problem Formulation
### Problem Formulation  
In this section, we first formulate a throughput optimization problem to maximize the sum of throughput of all users by selecting the optimal subframes and signal malicious jamming information on the interactive jammer userbase station channels for each user. We propose a collaborative decoupling distributed double Q-learning algorithm to solve this problem. Due to the interference between users and the malicious jamming of the external jammer, the communication between users will fail. We introduce a binary variable \(\xi_{n,m,k}\in\{0,1\}\). The binary variable represents the result of subframe-channel selection for the \(nth\) user. If \(nth\) user selects the \(mth\) channel in the \(kth\) subframe, then \(\xi_{n,m,k}=1\), and otherwise \(\xi_{n,m,k}=0\). The number of users occupying the \(mth\) channel in the \(kth\) subframe is defined as \(l_{m,k}=\sum_{n}\xi_{n,m,k}\). The users' set of successful transmission is \(S(n^{\prime})\triangleq\left\{n^{\prime}|\sum_{v}1(l_{m,k}=1)\xi_{n,m,k}=1\right\}\), where \(n^{\prime}\) is users that choose the idle channel; \(1(\cdot)\) indicates function. When the events of \((\cdot)\) is true, the \(1(\cdot)=1\). If user \(n\) occupies the \(mth\) idle channel in the \(kth\) subframe, then user \(n\) can successfully transmit the packet to the base station. Assume that the base station can only successfully receive users in user set \(S(n^{\prime})\). Then, a binary variable \(\beta_{n,m,k}\) is introduced to represent whether user \(n\) can make the base station successfully receive the packet by selecting the \(mth\) channel in the \(kth\) subframe. If \(n\in S(n^{\prime})\), then \(\beta_{n,m,k}=1\), otherwise, \(\beta_{n,m,k}=0\). The goal of the paper is to maximize the sum of throughput of all users. In this paper, it is assumed that the throughput is the ratio of the number of successful user communications to the number of subframes. The optimization problem can be formulated as follows:  
\[\max_{\{\beta_{n,m,k}\}}O=\sum_{n=1}^{N}\beta_{n,m,k}/K \tag{1}\]  
Figure 1: System model. s.t. \(\beta_{n,m,k}\in\{0,1\}\)  
\(\forall n\in[1,2,\ldots,N]\)  
\(\forall m\in[1,2,\ldots,M]\)  
\(\forall k\in[1,2,\ldots,K]\)  
According to the literature [23], Equation (1) is a non-convex function, so it is usually difficult to directly obtain the optimal solution. In order to maximize the system throughput, the channel conflicts between users should be minimized and the channel without malicious jamming should be selected. For each user, an idle channel should be selected in a subframe. Ideally, each user selects a unique subframe and channel at a time to send its packets. The selected channel is not jammed by jammers. However, this ideal is almost impossible to achieve with completely random access. However, access control methods based on machine learning (ML) algorithms (such as DQL algorithm) are expected to solve the problem (1). In this paper, the solution of Equation (1) can be found automatically by the distributed DQL algorithm deployed on all users, so that each user can find the optimal selection result of available subframes and channels, and avoid the interference between users and the malicious jamming of external jammers as much as possible. The process of joint channel and subframe selection is modeled as a two-element Markov problem, where each user is the learning subject. Its state space is the currently available channel subframe combination:  
\[\mathbf{S_{k,m}}=\{(k,m)\},k\in(1,2,\ldots,K),m\in(1,2,\ldots,M) \tag{2}\]  
The action space of channel and subframe joint selection Markov is defined as the joint action of channel and subframe that can be selected:  
\[\mathbf{A}_{k,m}=\{(k,m)\},k\in(1,2,\ldots,K),m\in(1,2,\ldots,M) \tag{3}\]  
The current reward function of user \(n\):  
\[R_{n}(k_{n},m_{n})= \begin{cases}1&(k_{n},m_{n})\
eq(k_{j},m_{j})\&\ (k_{n},m_{n})\
eq(k_{i},m_{i})(i\in N/n)\\ -1&\text{otherwise}\end{cases}  \tag{4}\]  
where, \((k_{n},m_{n})\) denotes the subframe and channel selected by user \(n\).