## 2 System Model
### The EES4-RLC Algorithm  
The algorithm begins with the clustering initialization phase which deals with clusterhead emergence as outlined in Table 1. All SUs (line 1 to line 20) sense a set of predefined channels (line 2), compute their clusterhead probabilities based on the number of vacant channels detected, percentage of clusterheads and their residual energy (line 3). If the clusterhead probability is greater or equal to the threshold, then a clusterhead announcement is broadcast (line 5) and the SU becomes a tentative clusterhead (line 6), otherwise, if the clusterhead probability is less than the threshold and it has heard the clusterhead announcement (line 7), then it waits for the final announcement (line 8) and the SU is a member node (line 9). Another set of SUs (line 10) perform iteration (line 11 to line 18). In each iteration,we increment the probability by a factor (line 12) and compare the result with the threshold (line 13). If it is clusterhead (line 15), it broadcasts an announcement (line 14). The Base Station selects an optimal number of clusterheads among the potential clusterheads (line 21) and broadcasts the list (line 24). The selected clusterheads broadcast advertisement packets to their neighboring SUs (line 24 to line 25). The RL process shown in Table 2 begins immediately after the emergence of clusterheads by setting all state-action values \(Q(|S|,|A|)\) array and action to zero (line 2) and then carrying out a number of iterations (line 3 to line 30) up to maximum episodes \(E_{ps}\) specified in the inputs (line 1). The state transition follows a sequential order (line 4) and is re-started upon reaching the number of clusters (line 28) _i.e._, number of elements in \(H_{n}\) (line 1). The softmax action selection strategy is employed (line 5) in each episode to select a clusterhead among the prospective clusterheads (line 7). This leads to sensing a set of channels \(|S_{ch}|\) (line 8) and computing the energy consumptions (line 9) and then obtaining the associated rewards (line 11 to line 25). The state-action Q-values (line 26) are updated by the cumulative reward (line 10) and the final Q-matrix is determined after the last episode. The state-action that returns the maximum total cumulative reward value (line 31) denotes the optimal policy \begin{table}
\begin{tabular}{l} \hline \hline \multicolumn{1}{l}{ Required: \(N,\mathbf{E_{n}^{Re}},\mathbf{\emph{q}}^{*},\mathbf{\psi},\mathbf{\emph{E_{n} }^{mx}},\mathbf{\acute{\xi}}}\) \\
1: **for \(\mathbf{\emph{n}\gets 1\emph{ to }N\emph{ do}}\)** \\
2: & Sense Channels (\(\mathbf{\emph{Ch}_{z}}|\mathbf{\emph{z}=1,2,3,...,n_{z}}\)) \\
3: & Compute \(\mathbf{\emph{p}_{n}^{ch}=\mathbf{E_{n}^{Re}}\)\(\mathbf{\emph{\varphi}_{n}\)\(\mathbf{\psi/E_{n}^{mx}}\)\(\mathbf{\emph{n}_{z}}\)}}\) \\
4: & **if (\(\mathbf{\emph{p}_{n}^{ch}\geq\mathbf{\psi}}\))** \\
5 & Broadcast \(\mathbf{\emph{A}_{pkt}}\) \\
6: & \(\mathbf{\emph{CH}=CH+1}\) \\
7: & **else if (\(\mathbf{\emph{p}_{n}^{ch}<\mathbf{\psi}}\))** and Received \(\mathbf{\emph{A}_{pkt}}\)** \\
8: & Wait for final clusterhead announcement \\
9: & \(\mathbf{\emph{MN}=MN+1}\) \\
10: & **else** \\
11: & \(\mathbf{\emph{for}\downarrow\leftarrow 1\emph{ to }N_{ir}\emph{ do}}\) \\
12: & \(\mathbf{\emph{p}_{n}^{ch}\leftarrow\mathbf{\emph{p}_{n}^{ch}+\mathbf{\acute{ \xi}}\)\(\mathbf{\psi}\)}}\) \\
13: & **if (\(\mathbf{\emph{p}_{n}^{ch}\geq\mathbf{\psi}}\))** then \\
14: & Broadcast \(\mathbf{\emph{A}_{pkt}}\) \\
15: & \(\mathbf{\emph{CH}=CH+1}\) \\
16: & Break \\
17: & **end if** \\
18: & **end for** \\
19: & **end if** \\
20: **end for** \\
21: & BS Selects \(\mathbf{\emph{q}}^{*}\) from \(\mathbf{\emph{CH}_{n}}|\mathbf{\emph{n}=1,2,3,...,CH}\) \\
22: & Broadcasts \(\mathbf{\emph{H}=\{CH_{n}}|\mathbf{\emph{n}=1,2,3,...,q}^{*}\)} \\
23: & **for \(\mathbf{\emph{j}=1\emph{ to }q^{*}\)do** \\
24: & Broadcast \(\mathbf{\emph{A}_{pkt}}\) \\
25: & **end for** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Initialization phase.