## 4 Deep Reinforcement Learning-Based Resource Allocation Policy
### Markov Decision Process
#### 4.1.2 Transmission Mode  
The global sensing decision indicates that the primary channel is free in the current time slot, and then, the CBS assigns transmission power levels to the CUs for transmitting their data to the CBS. We assume that the data of the CUs will be successfully decoded if the primary channel is actually free; otherwise, no data can be retrieved due to collisions between the signals of the PUs and CUs. In this case, there are two possible observations, as follows.  
_Observation 1_ (\(\Phi_{1}\)): All data are successfully received and decoded at the CBS at the end of the time slot. This result means the primary channel was actually free during this time slot, and the global sensing result was correct. The total reward for the network is calculated as  
\[R\left(s\left(t\right)\left|\Phi_{1}\right.\right)=\sum_{m=1}^{M}R_{m}\left(t \right), \tag{11}\]  
where the immediate data transmission rate of NOMA group \(G_{m}\), \(R_{m}\left(t\right)\), can be computed with Equation (4). Belief \(\mu\left(t+1\right)\) for the next time slot is updated as  
\[\mu\left(t+1\right)=P_{FF}. \tag{12}\]  
The remaining energy in \(CU_{im}\) for the next time slot will be  
\[e_{im}^{re}\left(t+1\right)=\min\left(e_{im}^{re}\left(t\right)+e_{im}^{h} \left(t\right)-e_{ss}-e_{im}^{tr}\left(t\right),E_{bat}\right), \tag{13}\]  
where \(e_{im}^{tr}\left(t\right)\) denotes the transmission energy assigned to \(CU_{im}\) in time slot \(t\).  
_Observation 2_ (\(\Phi_{2}\)): The CBS can not successfully decode the data from the CUs at the end of time slot \(t\) due to collisions between the signals of the CUs and the PUs. It implies that the primary channel was occupied, and misdetection happened. In this case, no reward is achieved, i.e., \(R\left(s\left(t\right)\left|\Phi_{2}\right.\right)=0\). Belief \(\mu\left(t+1\right)\) for the next time slot can be updated as  
\[\mu\left(t+1\right)=P_{BF}. \tag{14}\]  
The remaining energy in \(CU_{im}\) for the next time slot is updated by  
\[e_{im}^{re}\left(t+1\right)=\min\left(e_{im}^{re}\left(t\right)+e_{im}^{h} \left(t\right)-e_{ss}-e_{im}^{tr}\left(t\right),E_{bat}\right). \tag{15}\]  
In reinforcement learning, the agent is capable of improving the policy based on the recursive lookup table of state-value functions. The state-value function, \(V^{\pi}\left(s\right)\), is defined as the maximum expected value of the accumulated reward starting from current state \(s\) with the given policy, which is written as [28]:  
\[V^{\pi}\left(s\right)=E\left\{\sum_{t=1}^{\infty}\gamma^{t}R\left(t\right) \left|s\left(t\right)=s,\pi\right\}, \tag{16}\]  
where \(E\left\{.\right\}\) denotes the expectation, in which \(\gamma\in\left(0,1\right)\) is the discount factor, which can affect the agent's decisions on myopic or foresighted operations; \(\pi\) is the stochastic policy, which maps environment state space \(\mathbb{S}\) to action space \(\mathbb{A}\), \(\pi\left(a\left|s\right.\right)=\Pr\left(\boldsymbol{a}\left(t\right)=a\left| s\left(t\right)=s\right.\right)\). The objective of the resource allocation problem is to find optimal policy \(\pi^{*}\) that provides the maximum discounted value function in the long run, which can satisfy the Bellman equation as follows [41]:  
\[\pi^{*}\left(a\left|s\right.\right)=\operatorname*{arg\,max}_{\pi}V^{\pi}\left( s\right). \tag{17}\]The policy can be explored by using an \(\epsilon-greedy\) policy in which a random action is chosen with probability \(\epsilon\), or an action can be selected based on the current policy with probability \((1-\epsilon)\) during the training process [42]. As a result, the problem of joint power and bandwidth allocation in Equation (7) can be rewritten as Equation (17), and the solution to deep actor-critic reinforcement learning will be presented in the following section.