## 4 Joint Routing and Resource Management with Conjecture Based Multi-Agent Q-Learning
### Analysis of ERT-CMQL
: In accordance with the definition of the distance between Q-values, we have:  
\[\|P(Q)-P(Q^{\prime})\| = \underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}|P(Q_{i}(s_{i},a_{i}))-P(Q^{\prime}_{i}(s_{ i},a_{i}))| \tag{33}\] \[= \underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}|\underset{(a_{i},a_{-i})\in A}{\sum}R_{i}(s _{i},a_{i},a_{-i})\big{[}\widehat{q}_{i}(s_{i},a_{-i})-\widehat{q}^{\prime}_{i} (s_{i},a_{-i})\big{]}\] \[\leq \underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}\Big{[}\underset{(a_{i},a_{-i})\in A}{\sum}R_{i}(s _{i},a_{i},a_{-i})\big{[}\widehat{q}_{i}(s_{i},a_{-i})-\widehat{q}^{\prime}_{i} (s_{i},a_{-i})\big{]}\bigg{|}\] \[+\underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}\Big{[}\underset{b_{i}\in\mathbf{A}_{i}}{\max} \beta|Q_{i}(s^{\prime}_{i},b_{i})-Q^{\prime}_{i}(s^{\prime}_{i},b_{i})\big{]}\] \[= \underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}\Big{[}\underset{(a_{i},a_{-i})\in A}{\sum}R_{i}(s _{i},a_{i},a_{-i})\big{[}\widehat{q}_{i}(s_{i},a_{-i})-\widehat{q}^{\prime}_{i} (s_{i},a_{-i})\big{]}+\beta\|Q-Q^{\prime}\|\]  
where the second equation is derived from Equation (29), and the first inequality is obtained according to the transformation \(|A+B|\leq|A|+|B|\). It can be easily proven that \(|\text{max}A-\text{max}B|\leq\text{max}|A-B|\), so we can attain the second inequality. Since \(a_{i}\) is unrelatedto \(Q_{i}(s^{\prime}_{i},b_{i})-Q^{\prime}{}_{i}(s^{\prime}_{i},b_{i})\), \(\max\limits_{i\in M}\max\limits_{s_{i}\in\mathbf{S_{i}}}\max\limits_{a_{i}\in \mathbf{A_{i}}}\left[\max\limits_{b_{i}\in\mathbf{A_{i}}}\beta|Q_{i}(s^{\prime} _{i},b_{i})-Q^{\prime}{}_{i}(s^{\prime}_{i},b_{i})|\right]\) can be rewritten as \(\max\limits_{i\in M}\max\limits_{s_{i}\in\mathbf{S_{i}}}\max\limits_{b_{i} \in\mathbf{A_{i}}}\beta|Q_{i}(s^{\prime}_{i},b_{i})-Q^{\prime}{}_{i}(s^{\prime }_{i},b_{i})|\) and the last equation is attained using Equation (30). Next, we apply Equation (23) to the item \(\sum\limits_{(a_{i},a_{i},a_{-i})\in A}R_{i}(s_{i},a_{i},a_{-i})\left[\widetilde {q}_{i}(s_{i},a_{-i})-\widetilde{q}^{\prime}_{i}(s_{i},a_{-i})\right]\), and the expression can be rewritten as:  
\[\|Q-Q^{\prime}\|=\underset{i\in M}{\max}\underset{s_{i}\in\mathbf{S}_{i}}{\max} \underset{a_{i}\in\mathbf{A}_{i}}{\max}|Q_{i}(s_{i},a_{i})-{Q_{i}}^{\prime}(s_{i},a_{i})\big{|}. \tag{30}\]0  
which leads to:  
\[ \begin{array}{rcl}\|P(Q)-P(Q^{\prime})\|\leq&\max_{i\in M}\max_{s_{i}\in \mathbf{S_{i}}}\max_{a_{i}\in\mathbf{A_{i}}}\frac{1-\beta}{|\mathbf{A_{i}}|} \cdot\big{|}Q_{i}(s_{i},a_{i})-Q^{\prime}_{i}(s_{i},a_{i})|+\beta\|Q-Q^{\prime} \|\\ &\qquad\leq\frac{1-\beta}{v}\|Q-Q^{\prime}\|+\beta\|Q-Q^{\prime}\|=\frac{1- \beta+\beta v}{v}\|Q-Q^{\prime}\|\end{array}  \tag{42}\]  
where \(v=\underset{i\in N}{\min}|\mathbf{A_{i}}|>1\). Then we have \(v\cdot(1-\beta)>1-\beta\), so that \(v-\beta v>1-\beta\), which leads to \(\frac{1-\beta+\beta v}{v}<1\). Consequently, condition (2) is satisfied in the Lemma, and ERT-CMQL is proven to converge if \(\eta\) is large enough for all agents.