## 2 System Model
### Modelling of RL-Based Clustering
If \(Q_{k}^{t}=(s_{0}^{t},\,a_{0}^{t},s_{1}^{t},a_{1}^{t},...,s_{k-1}^{t},a_{k-1}^{t },s_{k}^{t},a_{k}^{t})\) denotes the sequence of state-actions executed from \(s_{0}^{t}\) to \(s_{k}^{t}\) in episodes \(t=\{0,1,2,...,\mathrm{T}\}\) and \(CH_{k-1}^{t}\) denotes the corresponding set of selected clusterheads for the state-actions then the action taken can be formulated based on a stochastic process as \(a_{k}^{t}=\sigma_{k}^{t}(Q_{k}^{t})\in(\mathcal{A}_{s_{k}^{t}})\), where \(\mathcal{A}_{s_{k}^{t}}=\mathcal{A}\backslash\{\{s_{k}^{t}\}\cup CH_{k-1}^{t}\), denotes a set of selected actions, \(\sigma_{k}^{t}\) denotes decision rule that maps the sequence of state-action \(Q_{k}^{t}\) into a probability distribution \(\Delta_{\sigma_{k}^{t}}(\mathcal{A}_{s_{k}^{t}})\)[3]. To achieve a policy \(\pi\) that maximizes long-term rewards, a [PERSON] distribution-based action selection strategy known as softmax is adopted to balance between exploration of random actions and exploitation of state-actions. The strategy selects an action that returns the highest estimated reward for the state-action values based on a probability \(\boldsymbol{\rho}\) determined by a positive parameter \(\tau^{t}\) called temperature. The probability \(\boldsymbol{\rho}\) can be expressed as [27]:  
\[\boldsymbol{\rho}(s_{k}^{t},a_{k}^{t}=j\,)=\frac{e^{Q(s_{k}^{t},a_{k}^{t}=j)/ \tau^{t}}}{\sum_{\overline{h}}^{\left|a_{\sigma_{k}^{t}}\right|}e^{Q(s_{k}^{t },a_{k}^{t}=\overline{h})/\tau^{t}}},j\in\mathcal{A}_{s_{k}^{t}} \tag{40}\]  
**Transition Probability**: The transition probability \(\mathcal{T}\colon S\times A\times\mathcal{S}\,...\,S\rightarrow[0,1]\) maps the state-action transit to a probability of moving from current state \(s_{k}^{t}\) to next state \(s_{k+1}^{t}\) whenever an action \(a_{k}^{t}\) is executed in the state \(s_{k}^{t}\). Therefore, the transition probability \(P_{m}(s_{k}^{t}/s_{k+1}^{t},a_{k}^{t})\) from current state \(s_{k}^{t}=v\) to the next state \(s_{k+1}^{t}=h\) is a function of the action \(a_{k}^{t}=j\) executed in the current state. However, the adopted learning algorithm which is Q-learning does not require transition probabilities. **Reward function:** Reward \(\mathcal{R}:S\times A\times S\,...\,S\rightarrow\,r_{k}^{t}\) is a key component of the MDP model that can be used to evaluate the state-action value for each episode and update the Q-table. It maps the state transition from state \(s_{k}^{t}\) to subsequent state \(s_{k+1}^{t}\) for the action \(a_{k}^{t}\) taken to an actual value reward [3]. In each episode, the agent selects an action \(a_{k}^{t}=\mathrm{j}\) to select clusterhead \(CH_{j}\) in state \(s_{k}^{t}=v\), computes its reward \(r_{k}^{t}\) before moving to the next state \(s_{k}^{t}=h\), and then restarts the state \(s_{k}^{t}=0\) upon reaching the last state. The expected cumulative reward \(r_{k+1}^{t}\) can be computed based on agent's reward for energy consumption \(rw_{E,k+1}^{t}\) and reward for local decision accuracy \(rw_{D,k+1}^{t}\) subject to the pairwise constraint condition \(P_{w,k+1}^{t}\). Thus:  
\[r_{k+1}^{t}\,(s_{k}^{t},a_{k}^{t})=\,\frac{1}{2}\big{[}\,rw_{E,k+1}^{t}+rw_{D,k+1}^{t}\,\big{]}I_{A}\big{\{}P_{w,k+1}^{t}\
eq 0\big{\}} \tag{41}\]  
where \(k\,=0,1,2,...\,,k^{t}-1\) denotes the stage index at episode \(t\), the indicator function \(I_{A}\{x\}\) can either be zero or one, depending on \(x\). The pairwise constraint condition \(P_{w,k+1}^{t}\) can be expressed as:  
\[P_{w,k+1}^{t}=\, \begin{cases}1,M_{v,i}\cap H_{V,j}\
eq\emptyset\\ 0,otherwise\end{cases}  \tag{42}\]  
This means that the cumulative reward will be zero if the member node \(MN_{i}\) and the chosen clusterhead \(CH_{j}\) share no common vacant channel \(M_{v,i}\cap H_{V,j}=\emptyset\). The reward for energy consumption \(rw_{E,k+1}^{t}\) can be obtained from the member node's \(MN_{i}\) consumed energy \(E_{tM}\) for transmitting \(B_{dt}\)bits of data packet to a neighboring clusterhead \(CH_{j}\) and for performing cooperative channel sensing \(E_{MN}^{cs}\).