## 2 System Model
Each SU senses a predefined set of channels to detect the presence or absence of PUs in the channels, computes its clusterhead probability \(p_{n}^{ch}\) based on the number of vacant channels detected \(\varphi_{n}\) and residual energy \(\mathrm{E}_{n}^{\mathrm{Re}}\), as well as the required percentage of clusterheads \(\psi\) (e.g., 5%) for the network. The clusterhead probability is the probability of a sensor node to become a clusterhead. The main goal is to ensure that all SUs in the network are covered by a set of clusterheads at the initial stage, so that member nodes can directly communicate with clusterheads within their radio range via a single-hop while the clusterheads communicate with the BS through a single-hop or multi-hop fashion. The clusterhead probability \(P_{ch}\) for secondary user \(SU_{n}\) can be expressed as:  
\[p_{n}^{ch}=\frac{\mathrm{E}_{n}^{\mathrm{Re}}\ \varphi_{n}\ \psi}{\mathrm{E}_{n}^{ \mathrm{mx}}\ n_{\mathrm{z}}} \tag{1}\]  
where \(\mathrm{E}_{n}^{\mathrm{mx}}\) denotes the reference maximum energy of the SU when fully charged. This clusterhead probability is similar to the HEED protocol described in [32]. The clustering initialization process begins at time \(\tau_{ini}\). Each SU determines its clusterhead probability and compares it with a given threshold \(\psi<1\). If its clusterhead probability is greater or equal to the given threshold \(p_{n}^{ch}\geq\psi\), then the SU emerges as a tentative clusterhead and then broadcasts an advertisement packet \(A_{pkt}\) comprising its ID and clusterhead probability. This means that SUs with the highest probability are more likely to emerge as the tentative clusterheads. SUs with clusterhead probability less than the threshold (\(p_{n}^{ch}<\psi\)) hearing the clusterheads announcement withdraw from competing and wait for the final clusterheads announcement by the BS. These SUs are more likely to remain as member nodes while the other in the set perform \(N_{tr}=1/\xi\) maximum number of iterations and compete for the role of clusterhead after expiration of announcement waiting period \(\sigma_{s}\). Where \(\xi\) denotes percentage of the threshold which can be set to a value less than one \(\xi<1\). In each iteration \(i_{tr}\), \(i_{tr}\leq N_{tr}\), each of the SUs increases its clusterhead probability \(P_{ch}\) by \(\xi\) ( e.g., \(\xi=10\%\) of \(\psi\) ) and compares the updated clusterhead probability with the given threshold. If the clusterhead probability is greater than or equal to the threshold, then it terminates the iteration and broadcasts an advertisement packet. Otherwise, it proceeds to the next iteration. The set of SUs with least number of iterations would emerge as the tentative clusterheads, then after expiration of announcement waiting period \(\sigma_{s}\) the other sets that follow them would be elected. This process continues until all nodes are covered by the clusterheads. This means that after the emergence of the first set of clusterheads, each subsequent emergence of sets of clusterheads would be delayed by some duration depending on the number of iterations. The BS selects an optimal number of clusterheads among the tentative clusterheads and broadcasts the list. The set-up phase mainly deals with cluster formation, based on the advertisement packet \(A_{pkt}\) received from multiple neighboring clusterheads \(CH_{j}|j=1,2,3,...,n_{hg}\) which also denotes clusters \(cl_{j}|j=1,2,3,...,n_{hg}\), hence clusterhead and cluster will be used interchangeably in this paper. The SU learns the energy consumed and local decision accuracy for each of the clusters by executing model-free reinforcement learning and then selects an optimal cluster \(cl_{j}^{*}\) that minimizes energy consumption and enhances spectrum holes detection.