## 2 System Model
### Modelling of RL-Based Clustering  
The problem of selecting optimal clusterhead is formulated as a Markov Decision Process (MDP) where a SU learns the energy consumption and local decision accuracy for neighboring clusterheads and then selects an optimal cluster that minimizes energy consumption and improves sensing performance. A quadruple \((\mathcal{S},\mathcal{T},\mathcal{A},\mathcal{R})\) represents the Markov Decision Process (MDP) for selecting an optimal cluster in the network, \(\mathcal{S}\) denotes set of states in the model of the operating environment \(\mathbf{s}=\{s_{1},s_{2},s_{3},s_{4}\...\,,s_{n}\},s\in\mathcal{S}\), \(\mathcal{T}\) denotes state transition function, \(\mathcal{A}\) denotes a set of actions to be executed \(a=\{a_{1},a_{2},a_{3}\...\,,a_{n}\}\), \(a\in\mathcal{A}\), while \(\mathcal{R}\) denotes the state reward function \(r(s,a)\in\mathcal{R}\)[19]. Each of the SUs or the agent selects an action \(a_{k}\) in every state \(s_{k}\) of the model as shown in Figure 5. The selected action \(a_{k}\) leads to sensing a set of channels, reporting local decisions to a clusterhead, computing the energy consumption and evaluating the local decision accuracy for the chosen cluster. Reward \(r_{k+1}\) obtained from the computed energy consumption and local decision accuracy for state \(s_{k}\) determines the next state \(s_{k+1}\) and the next action \(a_{k+1}\), \(k\) denotes the stage index of the process. The agent adopts an optimal policy \(\pi\) that maximizes the cumulative reward obtained from a known state experience and from exploitation of unknown states to select the optimal clusterhead. **States:** The state \(s_{k}\) of the MDP stands for the stage at which the agent selects a cluster among the neighboring clusterheads and determines the reward for taking an action \(a_{k}\) in the state. The set \(|S|\) comprises of states equal to the number of neighbouring clusterheads plus an initialization state which initiates the state transition for selecting the clusterhead. Initialization of the clusterhead selection process begins at \(k=0\), \(v=0\) for \(s_{k}^{t}=v\), where \(t=\{0,1,2,\ldots,T\}\) denotes the decision episode, \(k\) denotes the stage index and \(v\in\mathcal{S}\) denotes the current state number which indicates the selected clusterhead, if \(v\
eq 0\). Therefore, at every stage index \(k\
eq 0\) of the learning process, the state \(s_{k}^{t}=\ v\in\ \mathcal{S}\) for selecting an action \(a_{k}^{t}\in\ \mathcal{A}\) can be expressed as:  
\[s_{k}^{t}=v\.I_{x}(s) \tag{38}\]  
where \(I_{x}(s)\) is an indicator function in that:  
\[I_{x}(s)= \begin{cases}1,if\ a_{k}^{t}\
eq 0\\ 0,otherwise\end{cases}  \tag{39}\]  
For each episode \(t\) of the learning process, the agent \(i\) employs a softmax action selection strategy to select an action \(a_{k}^{t}\), computes the state-action reward \(r_{k}^{t}\in\mathcal{R}\) for the current state \(s_{k}^{t}=v\), and then determines the subsequent state \(s_{k+1}^{t}=h\in\mathcal{S}\). **Actions:** an action \(a_{k}\) in this context implies a strategic choice made by an agent for selecting a clusterhead \(CH_{j}\) among the neighbouring clusterheads \(CH_{j}|j=1,2,3,...\,,n_{hg}\). The selected action  
Figure 5: Model of reinforcement learning-based clustering. \(a_{k}^{t}=j\in\mathcal{A}\) in every state \(s_{k}^{t}=v\in\mathcal{S}\) is expected to maximize the current reward \(r_{k}^{t}\) for updating Q-value \(Q_{k}^{t}(s_{k}^{t},a_{k}^{t})\).