## 4 Performance Analysis and Evaluation  
In this section, we analyze optimality of the EESA-RLC algorithm and evaluate its performance in terms of convergence, complexity and adaptability to model a free dynamic environment in achieving an optimal policy \(\pi_{k}^{*}\) that minimizes energy consumption while enhancing vacant channel detection. The optimal policy \(\pi_{k}^{*}\) maximizes the cumulative reward \(r_{k}^{t}\) to achieve the optimal cluster \(cl_{j}^{*}\) by choosing an optimal clusterhead \(CH_{j}^{*}\) for the agent. \begin{table}
\begin{tabular}{l} \hline \hline
1: Input: \(|S|\), \(|A|\), \(|S_{ch}|\), \(E_{ps}|\), \(|H_{n}|\), \(n_{hg}\) \\ \hline
2: _Initialize_: \(Q(|S|,|A|)\)\(\leftarrow\) 0, \(k\)\(\leftarrow\) 0, \(a_{k}^{t}\)\(\leftarrow\) 0 \\
3: **for \(t\)\(\leftarrow\) 1 to \(E_{ps}\) do** \\
4: \(k\)\(\leftarrow\)\(k+1\) \\
5: \(a_{k}^{t}\)\(\leftarrow\)\(actionStrategy(s_{k}^{t},Q)\) \\
6: **if \(a_{k}^{t}\)\(\
eq\) 0** and \(k\)\(\leq\)\(n_{hg}\) then \\
7:  Select clusterhead (\(CH_{j},a_{k}^{t}=j\)) \\
8:  Sense Channel (\(|S_{ch}|\)) \\
9:  Compute \(E_{TCj}^{set}=\{E_{TCj}|j=1,2,3,...,n_{hg}\}\) for \(|\)\(H_{n}|\) \\
10:  Compute reward \(r_{k+1}^{t}\)\(\leftarrow\)\(Average(rw_{E,k+1}^{t},rw_{D,k+1}^{t})\) \\
11: **if \(E_{TCj}=\min(E_{TC}^{set})\)** then \\
12: \(rw_{E,k+1}^{t}\)\(\leftarrow\) 1 \\
13: **else if \(E_{TCj}<\max(E_{TC}^{set})\)** then \\
14: \(rw_{E,k+1}^{t}\)\(\leftarrow\) 0. 5 \\
15: **else** \\
16: \(rw_{E,k+1}^{t}\)\(\leftarrow\) 0 \\
17: **endif** \\
18:  Compare decision (\(LD_{i}\), \(CD_{j}\)) \\
19: **if \(LD_{i}\cap CD_{j}=\)\(n_{x}\)** then \\
20: \(rw_{D,k+1}^{t}\)\(\leftarrow\) 1 \\
21: **else if \(LD_{i}\cap CD_{j}\geq(n_{x}/2)\)** then \\
22: \(rw_{D,k+1}^{t}\)\(\leftarrow\) 0. \\
23: **else** \\
24: \(rw_{D,k+1}^{t}\)\(\leftarrow\) 0 \\
25: **endif** \\
26: **update \(Q_{k}^{t}\)\(\leftarrow\)\(Q_{k}^{t}\)\(+\)\(\alpha[r_{k+1}^{t}+\gamma max_{\alpha}(Q_{k+1}^{t})-Q_{k}^{t}]\)** \\
27: **else** \\
28: \(k\)\(=\) 0 \\
29: **endif** \\
30: **end for** \\
31: \(\left[H_{opt},I\right]\leftarrow\max\limits_{s_{k}^{t}\in S,a_{k}^{t}\in \omega}\left|\sum_{t=1}^{E_{ps}}r_{k}^{t}\left(s_{k}^{t},a_{k}^{t}\right)\right|\) \\
32: \(CH_{j}^{*}\)\(\leftarrow\)\(|I|\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Reinforcement learning clustering. In the simulation, we assume a moderate scale network consisting of 250 SUs uniformly distributed in a 90 m \(\times\) 90 m square area and five PUs randomly deployed in the network. Each PU can operate on one channel such that SUs can only access unused licensed channels for data transmission. Each channel might be free or occupied at any time depending on PU activity. We also consider low power wireless sensor nodes in computing the energy dissipation for spectrum sensing, reporting to clusterheads and data transmission. All SUs are homogenous and uniformly distributed in the network area, therefore in our simulation and mathematical modeling, all radio parameter and energy dissipation values for processing the received signal samples, tuning the detector's circuit to channel's bandwidth and running the electronics circuit are the same. The parameter values used for both analysis and simulation as indicated in Table 3 are either computed based on parameter values obtained from the indicated references or originated from the sources indicated _in situ_. The Q-learning and SARSA algorithms were implemented in MATLAB to evaluate the performance of the EESA-RLC algorithm for \(E_{ps}=5000\) episodes [19]. We set the step size for the exploration of state-action pairs and for learning rate update to \(\alpha^{k}=a/(b+k)\), while the discount factor is set to \(\gamma=0.9\). The discount factor determines the level priority given to future rewards. A factor of zero makes an agent consider only immediate rewards [48].