## 4 Joint Routing and Resource Management with Conjecture Based Multi-Agent Q-Learning
### Equal Reward Time-Slots Based Conjectural Multi-Agent Q-Learning
 \tag{24}\]  
Then, the conjecture belief is updated as follows:  
\[\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})=\widetilde{\varphi}_{i}^{t -1}(s_{i},\mathbf{a}_{-1})\cdot\frac{\zeta_{i}^{t-1}(s_{i},a_{i})}{\zeta_{i}^ {t}(s_{i},a_{i})}. \tag{25}\]  
Since \(\varphi_{i}^{t}(s_{i},\mathbf{a}_{-1})=\prod_{j\in M\setminus\{i\}}\zeta_{j} ^{t}(s_{j},a_{j})\approx\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})\), the updating rule in Equation (20) can be rewritten as:  
\[Q_{i}^{t+1}(s_{i},a_{i})=(1-\alpha)Q_{i}^{t}(s_{i},a_{i})+\alpha\left[\sum_{(a _{i},a_{-i})\in A}R_{i}(s_{i},a_{i},a_{-i})\widetilde{\varphi}_{i}^{t}(s_{i}, a_{-i})+\beta\underset{b_{i}\in\mathbf{A_{i}}}{\max}Q_{i}^{t}(s^{\prime}_{i},b_{i}) \right]. \tag{26}\]  
Equation (26) shows that every SU node only uses private strategy and limited information exchange with its previous nodes to update its Q-value. \(SU_{i}\) conjectures the mixed-strategies for other competing SUs on the basis of their variations in response to their own strategy. However, strong correlations exist between \(\zeta_{i}^{t}(s_{i},a_{i})\) and \(\zeta_{i}^{t-1}(s_{i},a_{i})\), which may cause the parameters to easily stick in a poor local optimum and then make \(\zeta_{i}^{t-1}(s_{i},a_{i})/\zeta_{i}^{t}(s_{i},a_{i})\) close to \(1\) infinitely. Since the updating rule of \(\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})\) is fractional which has a strong reliance on \(\zeta_{i}^{t-1}(s_{i},a_{i})/\zeta_{i}^{t}(s_{i},a_{i})\), the conjecture belief is also inclined to fall into the local optimal solution. To avoid the shortage and further improve the system performance, experience replay is applied to the conjecture based multi-agent learning scheme. From long-term-observations, \(\overline{n}\) is a constant value due to the time stationarity of \(n\). So the probability that \(SU_{i}\) receives an expected reward \(R_{i}(s_{i},a_{i},\mathbf{a}_{-1})\) (i.e., agents perform action vector \((a_{i},\mathbf{a}_{-1})\) in state \(s_{i}\)) is approximately equal to the reciprocal of mean time interval regardless of time step, that is:  
\[\zeta_{i}^{t}(s_{i},a_{i})\cdot\varphi_{i}^{t}(s_{i},\mathbf{a}_{-1})\approx \frac{1}{1+\overline{n}}\approx\zeta_{i}^{v}(s_{i},a_{i})\cdot\varphi_{i}^{v} (s_{i},\mathbf{a}_{-1}), \tag{27}\]  
where \(t\) and \(v\) represent any two time slots. Thus we have \(\zeta_{i}^{t}(s_{i},a_{i})\cdot\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_ {-1})=\zeta_{i}^{v}(s_{i},a_{i})\cdot\widetilde{\varphi}_{i}^{v}(s_{i}, \mathbf{a}_{-1})\). \(\zeta_{i}(s_{i},a_{i})\) and \(\widetilde{\varphi}_{i}(s_{i},\mathbf{a}_{-1})\) at each time step are stored as the agent's experience at each time slot, and pooled over many episodes into a replay memory [27]. During learning, we randomly sample the experience \(\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})\) and \(\zeta_{i}^{k}(s_{i},a_{i})\) from memory pool to update the conjecture. Then the update of conjecture belief at time step \(t\) is given by:  
\[\widetilde{\varphi}_{i}^{t}(s_{i},\mathbf{a}_{-1})=\widetilde{\varphi}_{i}^{v} (s_{i},\mathbf{a}_{-1})\cdot\frac{\zeta_{i}^{v}(s_{i},a_{i})}{\zeta_{i}^{t}(s_ {i},a_{i})}. \tag{28}\]This approach has several advantages over consecutive updating rule in Equation (25). First, each time-step of the strategy is potentially used in the update of conjecture, which improves data efficiency instead of updating directly from consecutive samples. Second, strong correlations between \(\zeta_{i}^{t}(s_{i},a_{i})\) and \(\zeta_{i}^{t-1}(s_{i},a_{i})\) may result in a local optimal. Randomizing the samples breaks these correlations and reduces the variance in the updates. The details of ERT-CMQL are obtained as described in Algorithm 1. ```
1:Initialize:
2: Set \(t=0\) and memory size \(N\).