## 2 System Model  
The system considered in this paper comprises a pair of licensed primary users, several secondary transmitters (denoted as SUs), a secondary receiver equipped with a fusion center, and an eavesdropper (E), as shown in Figure 1. From now on, we will call the secondary receiver as the fusion center or FC for simplicity. In this work, the SUs are assumed to always have data to transmit to the fusion center. Thus, they would try to access the licensed channel of the PUs for data transmission by carrying out cooperative spectrum sensing. \begin{table}
\begin{tabular}{r l} \hline \hline
**Symbol** & **Description** \\ \hline \(s(t)\) & Primary user (PU) signal at time step \(t\) \\ \(x_{i}(t),w_{i}(t)\) & The received signal and additive white Gaussian noise at the \(i\)th secondary user (SU) at \\  & time step \(t\) \\ \(\mu\) & Mean \\ \(\sigma^{2}\) & Variance \\ \(\gamma_{i}\) & Average gain of the sensed channel at the \(i\)th SU \\ \hline \(E_{ca}\) & Battery capacity of the SU \\ \(E_{s}\) & The energy consumption for spectrum sensing process \\ \(e_{r},e_{b},e_{tr}\) & The remaining energy, harvested energy, transmit energy of the SU respectively \\ \(Nk\) & The key length of the encryption method \\ \(S_{Nk}\) & The security level corresponding with the key length \(Nk\) \\ \(P_{d},P_{f}\) & The system probabilities of detection and false alarm \\ \hline \(\langle\mathbb{S},\mathbb{A},\mathbb{P},\mathbb{R}\rangle\) & Markov decision process (MDP) Tuple: State space \(\mathbb{S}\), Action space \(\mathbb{A}\), Transition probability \\  & function \(\mathbb{P}\), and Reward function \(\mathbb{R}\) \\ \(\rho\) & The belief that the PU is inactive in a time slot \\ \(\eta\) & Discount factor \\ \(V(s)\) & State-value function \\ \(\pi(s)\) & Policy function \\ \(\alpha_{a},\alpha_{c}\) & The actor and the critic step-size parameters \\ \(\delta\) & Temporal difference (TD) error \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The list of the most used notations. Figure 1: The model of the system considered in this paper. E: eavesdropper; FC: fusion center. The primary user's states (active \([A]\) and not active \([\bar{A}]\)) are assumed to follow a two-state Markov discrete-time process, in which the transition probabilities between the states are denoted \(P_{i,j}:i,j\in\{A,\bar{A}\}\), as illustrated in Figure 2.  
The performance of the sensing scheme can be evaluated by using the probability of correct detection \(P_{d}\) and the probability of false alarm \(P_{f}\). The former represents the probability of detecting the _active_ state \((A)\) of the PU accurately, whereas the latter indicates the probability that the PU is identified as active, but it is truly not \((\bar{A})\), each of which are given by  
\[P_{d}=\Pr(H=A|A) \tag{1}\]  
and  
\[P_{f}=\Pr\left(H=A|\bar{A}\right), \tag{2}\]  
respectively, where \(H\) denotes the state of the primary user as determined by spectrum sensing. Although the PU state transition probabilities are unknown in practical situations, the historical statistics information of the primary channel can be used to estimate the state transition probabilities based on the Markov model [26]. Therefore, we assume that the SU has a prior information about the PU state transition probabilities based on the historical sensing results; and the global information of the network (e.g., channel state information, probabilities of detection and false alarm) are available for all nodes in the network. The system's operation proceeds as follows. The system is assumed to operate in a time-slotted manner. At the beginning of each time slot, the SUs perform spectrum sensing separately and send the sensing outcomes to the fusion center, where the data are fused together using a certain rule to decide the state of the primary user. The final sensing result is then broadcast to the SUs. If the channel is free, it is allocated to one of the SUs for data transmission. The SUs take turns using the channel, based on the arrival order of their transmission requests.