## 4 Reinforcement Learning for Improving Performance
The Cross model [22] is now widely recognized as one of the Reinforcement Learning models with memory-less characteristics, which means the learning process is a Markov Decision Process (MDP). The basic idea is to follow the rules of \"Results\" [23], namely, if system is rewarded by choosing a strategy, then the next period will get higher probability of choosing such strategy. On the contrary, if it is punished, the next period will reduce the probability of choosing such strategy. [PERSON] and [PERSON] [24] introduced the Bush-Mosteller model in 1955 [25]. Afterwards, [PERSON] and [PERSON] improved this model and introduced the Roth-Ever model. Nowadays, as two models of reinforcement learning, both of them [26] are widely adopted. They are easy to realize and have very low computation complexity, which fit for the real-time applications. Therefore, in this paper, these two models are introduced and some necessary modifications are adopted for the application, so the model of MDP Cross and Statistical Mean are proposed. As mentioned above, the process of connection setup is defined as the time-slotted. The optional strategies for the SUs are defined as follows:  
\[\mathbf{A}_{su}=\left(\Gamma_{1},\Gamma_{2},\ldots,\Gamma_{n},\ldots,\Gamma_{ n^{\prime}},\Gamma_{R}\right) \tag{49}\]  
where \(\mathbf{A}_{su}\) is the vector of optional strategies, \(R\) the number of the strategy, \(n\) is the chosen strategy and \(n^{\prime}\) are not chosen strategies in a certain time-slot. Consequently, during the time-slot \(k\) to access the initial stage, the SUs can update the probability of choosing strategy \(n\) and \(n^{\prime}\) by the following formula:\[ \begin{array}{ll}p^{n}\left(k+1\right)=p^{n}\left(k\right)+R\left[u\left(k\right) \right]\times\left(1-p^{n}\right)&n=A_{su}\left(k\right)\\ p^{n^{\prime}}\left(k+1\right)=p^{n^{\prime}}\left(k\right)-R\left[u\left(k \right)\right]\times p^{n^{\prime}}\left(k\right)&n^{\prime}\
eq A_{su} \left(k\right)\\ R\left[u\left(k\right)\right]=\alpha\times u\left(k\right)+\beta\end{array}  \tag{50}\]  
where \(A_{su}(k)\) is the accessible strategy of the SUs at the time-slot \(k\), which can be seen the action of MDP. \(p^{n}(k)\) is the probability of the accessible strategy \(n\) of the SUs at time-slot \(k\), \(p^{n}(k)\) is the probability of the unused strategy \(n^{\prime}\) of the SUs at time-slot \(k\), which can be seen as the state of MDP. \(u(k)\) is the reward function of the accessible performance of the SUs, which can be seen as the reward of MDP. \(\alpha\) and \(\beta\) are the adjustment factors, which can be used to determine the updating rate of \(u(k)\). \(R[u(k)]\) is defined as the monotone function of \(u(k)\), which is \(-1<R[u(k)]<1\). When the SUTCH successfully accesses idle subchannels, it obtains the reward, which is defined as follows:  
\[\partial_{1}I\left(k\right)C_{\text{SUTCH}}\left(k\right)T\left(k\right) \tag{51}\]  
where, \(T(k)\) is the transmission duration of the SUs in time-slot \(k\) and \(\partial_{1}\) is a weighting factor and \(I(k)\) is indicator function, which is defined as follows:  
\[\left\{ \begin{array}{ll}I\left(k\right)=1&\text{SUTCH successfully access at time-slot $k$}\\ I\left(k\right)=0&\text{SUTCH fail to access at time-slot $k$}\end{array} \right. \tag{52}\]  
When the SUTCH fails to access the idle subchannels, it wastes the opportunity for transmission and pays the cost, which is shown as follows:  
\[-\partial_{2}I\left(k\right)C_{\text{SUTCH}}\left(k\right)T^{\prime}\left(k\right) \tag{53}\]  
where, \(T^{\prime}(k)\) is the access duration of the SUs and \(\partial_{2}\) is also a weighting factor. Equivalently:  
\[u\left(k\right)=\partial_{1}I\left(k\right)C_{\text{SUTCH}}\left(k\right)T \left(k\right)-\partial_{2}I^{\prime}\left(k\right)C_{\text{SUTCH}}\left(k \right)T^{\prime}\left(k\right)\quad 0\leq\partial_{i}\leq 1,i=1,2 \tag{54}\]  
In order to weaken the impact of weighting on updating the probability of the choosing strategy, Equation (52) can be further defined as follows:  
\[