SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(V _ {\star} ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} Q _ {\star} ^ {s _ {t}} y _ {t} ^ {s _ {t}}\right) \right] \\ = \mathbb {E} \left[ \sum_ {t = t _ {i}} ^ {\infty} \mathbf {1} [ E _ {t} = 1 ] \left(V _ {\star} ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} G ^ {s _ {t}} y _ {t} ^ {s _ {t}} - \gamma V _ {\star} ^ {s _ {t + 1}}\right) \right] \\ = \mathbb {E} \left[ \sum_ {t = t _ {i}} ^ {\infty} \mathbf {1} [ E _ {t} = 1 ] \left(V _ {\star} ^ {s _ {t}} - x _ {t} ^ {s _ {t} ^ {\top}} G ^ {s _ {t}} y _ {t} ^ {s _ {t}} - \mathbf {1} [ E _ {t + 1} = 1 ] V _ {\star} ^ {s _ {t + 1}}\right) \right] \\ = \mathbb {E} \left[ V _ {\star} ^ {s _ {t _ {i}}} \right] - \mathbb {E} \left[ \sum_ {t = t _ {i}} ^ {\infty} \mathbf {1} [ E _ {t} = 1 ] x _ {t} ^ {s _ {t} ^ {\top}} G ^ {s _ {t}} y _ {t} ^ {s _ {t}} \right] \\ = \mathbb {E} _ {s \sim \rho} \left[ V _ {\star} ^ {s} \right] - \mathbb {E} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} x _ {t} ^ {s _ {t} ^ {\top}} G ^ {s _ {t}} y _ {t} ^ {s _ {t}} \right]. \\ \end{array}
$$
Combining this with Eq. (33), we get
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} x _ {t} ^ {s _ {t} ^ {\top}} G ^ {s _ {t}} y _ {t} ^ {s _ {t}} \right] \geq (\# \text {e p i s o d e s i n T s t e p s}) \mathbb {E} _ {s \sim \rho} [ V _ {\star} ^ {s} ] - o (T) \\ \geq (1 - \gamma) \mathbb {E} _ {s \sim \rho} \left[ V _ {\star} ^ {s} \right] T - o (T). \\ \end{array}
$$
Hence the one-step average reward is at least  $(1 - \gamma)\mathbb{E}_{s\sim \rho}[V_{\star}^{s}]$ . A symmetric analysis shows that it is also at most  $(1 - \gamma)\mathbb{E}_{s\sim \rho}[V_{\star}^{s}]$ . This shows that both players have no regret compared to the game value. Notice that this is only a loose implication of the path convergence guarantee because of the loose second inequality in Eq. (33).
Remark on the notion of "last-iterate convergence" in general Markov games While Eq. (31) corresponds to best-iterate convergence for general Markov games, an even stronger notion one can
follows the following two cases. In the first case, we assume that the policy is "last-iterate convergence." As argued above, it is impossible to require that the policies on all states to converge to equilibrium. To address this issue, we propose to study this problem under the episodic setting described above, in which the state is reset after every trajectory whose expected length is  $\frac{1}{1 - \gamma}$ . In this case, last-iterate convergence will be defined as the convergence of the following quantity to zero when  $i \to \infty$ :
$$
\mathbb {E} _ {s \sim \rho} \left[ \max _ {x, y} \left(V _ {x _ {t _ {i}}, y} ^ {s} - V _ {x, y _ {t _ {i}}} ^ {s}\right) \right]
$$
where we recall that  $i$  is the episode index and  $(x_{t_i}, y_{t_i})$  are the policies used by the two players at the beginning of episode  $i$ . While last-iterate convergence seems reasonable and possibly achievable, we are unaware of such results even for the degenerated case of single-player MDPs — the standard regret bound corresponds to best-iterate convergence, while the techniques we are aware of to prove last-iterate convergence in MDPs require additional assumptions on the dynamics.