## 4 Joint Routing and Resource Management with Conjecture Based Multi-Agent Q-Learning
### Analysis of ERT-CMQL  
[PERSON] [28] provided the convergence proof of the standard Q-learning. Based on the theory, the convergence of ERT-CMQL is investigated in this section. **Lemma. ** Suppose there is a mapping \(P:\mathbf{Q}\rightarrow\mathbf{Q}\), and \(\mathbf{Q}\) denotes the set of all SUs' Q functions. The updating rule \(Q^{t+1}=(1-\alpha)Q^{t}+\alpha\cdot P\big{(}Q^{t}\big{)}\) converges to \(Q^{*}\) with probability of 1, if:  
1. \(Q^{*}=E[P(Q^{*})]\)
2. A number \(0<\sigma<1\) exists such that \(\|P\big{(}Q^{t}\big{)}-P(Q^{*})\|\leq\sigma\|Q^{t}-Q^{*}\|\) for all \(Q^{t}\in Q\). To apply the lemma in the convergence proof of our proposed ERT-CMQL, the definition is given as follows:  
**Definition**. : _Let \(Q^{t}=\left(Q^{t}_{1},\ldots,Q^{t}_{M}\right)\), where \(Q^{t}_{i}\in\mathbf{Q}_{i}\) for \(i\in M\), and \(\mathbf{Q}=\prod_{i\in M}Q_{i}\). Then the mapping \(P:\mathbf{Q}\rightarrow\mathbf{Q}\) is defined as \(P\big{(}Q^{t}\big{)}=\big{[}P\big{(}Q^{t}_{1}\big{)},\ldots,P\big{(}Q^{t}_{M} \big{)}\big{]}\), where:_  
\[P\big{(}Q^{t}_{i}(s_{i},a_{i})\big{)}=\sum_{(a_{i},a_{i})\in A}R_{i}(s_{i},a_{ i},a_{-i})\widehat{\varphi}^{t}_{i}(s_{i},a_{-i})+\beta\underset{b_{i}\in\mathbf{A}_{i}}{ \max}Q^{t}_{i}(s^{\prime}_{i},b_{i}). \tag{29}\]  
_In addition, for any \(Q,\ Q^{\prime}\in\mathbf{Q}\), the definition of the distance between \(Q\)-values is given as:_  
\[\|Q-Q^{\prime}\|=\underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}|Q_{i}(s_{i},a_{i})-{Q_{i}}^{\prime}(s_{i},a_{i})\big{|}. \tag{30}\]  
Firstly, we prove the first condition in Lemma 1 for ERT-CMQL. **Proposition 1**. : \(Q^{*}\) _is equal to the expectation of its map \(P(Q^{*})\), i.e., \(Q^{*}=E[P(Q^{*})]\), where \(Q^{*}=\left(Q^{*}_{1},\ldots,Q^{*}_{M}\right)\)._  
Proof. : According to the [PERSON]'s optimality equation [29], we have the following expression:  
\[Q^{*}_{i}\left(s_{i},a_{i}\right)=E\big{[}R_{i}(s_{i},a_{i},\mathcal{L}^{*}_{ -i})\big{]}+\beta\underset{s^{\prime}_{i}\in S_{i}}{\sum}P_{s_{i}s^{\prime}_ {i}}(a_{i},\zeta^{*}_{-i})\underset{b_{i}\in A_{i}}{\max}Q^{*}_{i}(s^{\prime} _{i},b_{i}), \tag{31}\]  
Since the reward \(R_{i}(s_{i},a_{i},\mathcal{L}^{*}_{-i})\) is irrelevant to \(s^{\prime}_{i}\), then Equation (31) can be modified as:  
\[Q^{*}_{i}(s_{i},a_{i}) =\underset{s^{\prime}_{i}\in S_{i}}{\sum}P_{s_{i}s^{\prime}_{i}} (a_{i},\zeta^{*}_{-i})\bigg{\{}E\big{[}R_{i}(s_{i},a_{i},\zeta^{*}_{-i})\big{]} +\beta\underset{b_{i}\in A_{i}}{\max}Q^{*}_{i}(s^{\prime}_{i},b_{i})\bigg{\}} \tag{32}\] \[=\underset{s^{\prime}_{i}\in S_{i}}{\sum}P_{s_{i}s^{\prime}_{i}} (a_{i},\zeta^{*}_{-i})\bigg{\{}\underset{(a_{i},a_{-i})\in A}{\sum}R_{i}(s_{i},a_{i},a_{-i})\underset{j\in M\setminus\{i\}}{\prod}\zeta_{j}(s_{j},a_{j})+ \beta\underset{b_{i}\in A_{i}}{\max}Q^{*}_{i}(s^{\prime}_{i},b_{i})\bigg{\}}\]  
Based on the previous analysis in Section 4.2, we have \(\varphi^{t}_{1}(s_{i},\mathbf{a}_{-i})=\prod_{j\in M\setminus\{i\}}\zeta^{t} _{j}(s_{j},a_{j})\). Then we prove that \(Q^{*}_{i}(s_{i},a_{i})=E\big{[}P\big{(}Q^{*}_{i}(s_{i},a_{i})\big{)}\big{]}\). **Proposition 2**. : _There is a number \(0<\sigma<1\) such that \(\|P\big{(}Q^{t}\big{)}-P(Q^{*})\|\leq\sigma\|Q^{t}-Q^{*}\|\)._  
Proof.