## 4 Joint Routing and Resource Management with Conjecture Based Multi-Agent Q-Learning
### Equal Reward Time-Slots Based Conjectural Multi-Agent Q-Learning  
Multi-agent learning strategy is more practical for solving the joint design problem in multi-hop CRN. The main drawback of establishing a multi-agent learning framework is the demand for complete information of competing SUs. Due to high communication overhead and topology complexity, it is impractical for SU nodes to cooperate with competing SUs and share their private information in multi-hop CRN. To resolve this contradiction, a conjecture-based multi-agent learning scheme with quasi-cooperative scenario is proposed, where each SU node conjectures other SUs' behavior strategies without full coordination among agents. Specifically, from Equations (20) and (21), the mixed-strategies for other competing SUs is defined as \(\varphi_{i}^{t}(s_{i},\mathbf{a}_{-\mathbf{i}})=\prod_{j\in M\backslash\{i \}}\zeta_{j}^{t}(s_{j},a_{j})\), which represents the joint probability that competing SUs perform strategy vector \(\zeta_{-\mathbf{i}}=\left\{\zeta_{j}^{t}(s_{j},a_{j})\right\}_{j\in M \backslash\{i\}}\) at time slot \(t\). In other words, estimating \(\varphi_{i}^{t}(s_{i},\mathbf{a}_{-\mathbf{i}})\) becomes the key challenge when applying the multi-agent Q-learning framework. To combat this, the conjecture belief \(\widehat{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-\mathbf{i}})\) is introduced to approximate \(\varphi_{i}^{t}(s_{i},\mathbf{a}_{-\mathbf{i}})\), and the ERT-CMQL is proposed to asymptotically determine \(\widehat{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-\mathbf{i}})\) without complete network information. The probability that the agent chooses \(a_{i}\) in state \(s_{i}\) while other competing SUs execute action vector \(\mathbf{a}_{-\mathbf{i}}\) is given by:  
\[\delta_{i}=\zeta_{i}^{t}(s_{i},a_{i})\cdot\varphi_{i}^{t}(s_{i},\mathbf{a}_{- \mathbf{i}}). \tag{22}\]  
\(SU_{i}\) receives expected reward \(R_{i}(s_{i},a_{i},\mathbf{a}_{-\mathbf{i}})\) when the agent of \(SU_{i}\) performs action \(a_{i}\), while other nodes select action vector \(\mathbf{a}_{-\mathbf{i}}\) in state \(s_{i}\). That is, the probability that \(SU_{i}\) acquires \(R_{i}(s_{i},a_{i},\mathbf{a}_{-\mathbf{i}})\) is \(\delta_{i}\). \(n\) is the number of time steps between any two moments in which \(SU_{i}\) achieves the same return \(R_{i}(s_{i},a_{i},\mathbf{a}_{-\mathbf{i}})\). Each \(n\) is independent of the others and follows the same distribution of \(\delta_{i}\). The average value of \(n\) is denoted as \(\overline{n}\), which can be obtained via the private information from historical observation. Then we have the approximate equation \(\delta_{i}\approx 1/(1+\overline{n})\)[15], i.e., \(\zeta_{i}^{t}(s_{i},a_{i})\cdot\varphi_{i}^{t}(s_{i},\mathbf{a}_{-1})\approx 1 /(1+\overline{n})\). Since every SU knows its own transmission strategy \(\zeta_{i}^{t}(s_{i},a_{i})\), the agent can estimate \(\varphi_{i}^{t}(s_{i},\mathbf{a}_{-1})\) via:  
\[\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})=\frac{1}{(1+\overline{n}) \cdot\zeta_{i}^{t}(s_{i},a_{i})}. \tag{23}\]  
After obtaining the expression of \(\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})\) using local information shown in Equation (23), the updating rule of the conjecture belief is explored. In quasi-cooperative learning scenarios, agents update their conjecture belief based on new observations. Since \(n\) is a stationary stochastic process in the time dimension, its mean value \(\overline{n}\) is a constant. Specifically, the quotient of the conjecture belief at time slot \(t-1\) and \(t\) can be calculated as:  
\[ \begin{array}{cc}\frac{\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})}{ \widetilde{\varphi}_{i}^{t-1}(s_{i},\mathbf{a}_{-1})}&=\frac{1}{(1+\overline{n })\cdot\zeta_{i}^{t}(s_{i},a_{i})}\Bigg{/}\Bigg{[}\frac{1}{(1+\overline{n}) \cdot\zeta_{i}^{t-1}(s_{i},a_{i})}\Bigg{]}\\ &=\frac{\zeta_{i}^{t-1}(s_{i},a_{i})}{\zeta_{i}^{t}(s_{i},a_{i})}\end{array}