## 5 Simulation Results  
In this section, the performance of our quasi-cooperative multi-agent learning scheme is evaluated using an event-driven simulator coded in Python 3.5. The network model and learning framework are built based on the Python packages Networkx and Numpy, respectively. The results of the proposed ERT-CMAQL are compared with (1) Cooperative Multi-Agent Q-Learning (CMAQL) which is the ideal scheme and has complete information of the competing SUs; (2) Conjectural Multi-Agent Q-Learning without Experience Replay (CMAQL-ER); (3) Fixed Power- based Conjectural Multi-Agent Q-Learning (FP-CMAQL) proposed in [16] which transmits data with a fixed power level; (4) a single agent Q-learning scheme called Q-routing presented in [14] and (5) Prioritized Memories Deep Q-Network (PM-DQN) based joint design scheme proposed in our previous work [17]. In the multi-agent learning framework, we initialize the conjecture belief \(\widehat{\varphi}_{i}^{0}(s_{i},\mathbf{a_{-i}})=1\), the Q-value \(Q_{i}^{0}(s_{i},a_{i})=0\), and the transmission strategy \(\zeta_{i}^{0}(s_{i},a_{i})=1/|\mathbf{A_{i}}|\) for each \(s_{i}\in S_{i}\), \(a_{i}\in A_{i}\). Other system parameters are given in Table 2.  
To verify the performance of algorithms, a small CRN containing 10 SUs and 4 PUs is simulated at first. SUs in CRN are uniformly deployed in a 300 \(\times\) 300 m region. In addition, the available \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Link Gain & \(h=\varepsilon G(r/r_{0})^{-m}\), for \(r>r_{0}\)[15] \\ Available Spectrum & 56 MHz \(-\) 62 MHz \\ Bandwidth, \(B\) & 1 MHz \\ AWGN power, \(\hat{\sigma}\) & \(10^{-7}\) mW \\ PU-to-SU interference, \(\hat{\varphi}_{ijc}^{PU}\) & \(\hat{\varphi}_{ijc}^{PU}\sim\left[10^{-7},\,10^{-6}\right]\) mW \\ Packet Size, \(R_{packet}\) & \(2\times 10^{5}\) bit \\ Mean of PU Departure Rate, \(\mu\) & 0.1 \\ Deviation of PU Departure Rate, \(\sigma\) & 0.05 \\ SINR threshold, \(\gamma_{th}\) & 60 dBm \\ Outage probability, \(P_{ij}^{out}\) & \(P_{ij}^{out}\sim\left.N(0.1,\,0.05)\right.\) \\ Data flow arrival rate, \(\lambda_{f}\) & 0.4 \\ Data flow service rate, \(\mu_{f}\) & 0.6 \\ Maximal transmission delay, \(\kappa_{th}\) & 200 ms \\ Discount factor, \(\beta\) & 0.9 \\ Learning rate, \(\alpha\) & 0.1 \\ Time step size, \(t\) & 500 ms \\ Temperature, \(\eta\) & 0.005 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: System Parameters. transmitting power consists of five levels: \(\{50,\ 100,\ \dots,\ 250\ \mathrm{mW}\}\). The network topology is shown in Figure 2.  
Without loss of generality, SU 6 is taken as an example. The single-node performance of SU 6 for different algorithms is shown in Figures 3 and 4. Figure 3 illustrates the average reward of SU 6 versus the iteration index. The expected reward firstly rises and then stays almost steady for all schemes. Furthermore, we find that, when converged, CMAQL outperforms all other algorithms. The reward of ERT-CMAQL is slightly lower than CMAQL, followed by the CMAQL-ER scheme, and FP-CMAQL obtains the lowest reward. This occurs mainly because, in the CMAQL scheme, agents have true strategies of competing SUs through global information exchange. In ERT-CMAQL, each agent approximates mixed-strategies of other SUs via the conjecture belief that may be not sufficiently accurate. We can see that the reward of CMAQL-ER is slightly higher than that of ERT-CMAQL before 200 iterations, and afterward, that ERT-CMAQL is superior to CMAQL-ER. The reason for this is that at first few samples are stored in replay memory and the correlation is weak between the samples, so that experience replay is inefficient compared to the consecutive updating rule. At the later stage the advantage of experience replay is fully demonstrated when samples are abundant. In addition, FP-CMAQL obtains the lowest average reward, which illustrates the importance of power allocation. Figure 3: Expected reward of the SU 6 versus the iteration index. Figure 2: Network topology consisting of 10 SUs and 4 PUs.