Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning

## Abstract

Abstract Standard multi-agent reinforcement learning (MARL) algorithms are vulnerable to sim-to-real gaps. To address this, distributionally robust Markov games (RMGs) have been proposed to enhance robustness in MARL by optimizing the worst-case performance when game dynamics shift within a prescribed uncertainty set. Solving RMGs remains under-explored, from problem formulation to the development of sample-efficient algorithms. A notorious yet open challenge is if RMGs can escape the curse of multiagency, where the sample complexity scales exponentially with the number of agents. In this work, we propose a natural class of RMGs where the uncertainty set of each agent is shaped by both the environment and other agents’ strategies in a best-response manner. We first establish the well-posedness of these RMGs by proving the existence of game-theoretic solutions such as robust Nash equilibria and coarse correlated equilibria (CCE). Assuming access to a generative model, we then introduce a sample-efficient algorithm for learning the CCE whose sample complexity scales polynomially with all relevant parameters. To the best of our knowledge, this is the first algorithm to break the curse of multiagency for RMGs.

## 1 Introduction

A flurry of problems naturally involve decision-making among multiple players with strategic objectives. Multi-agent reinforcement learning (MARL) serves as a powerful framework to address these challenges, demonstrating potential in various applications such as social dilemmas (Leibo et al.,, ; Baker,, ; Zhang et al.,, ), autonomous driving (Lillicrap et al.,, ), robotics (Kober et al.,, ; Rusu et al.,, ), and games (Mnih et al.,, ; Vinyals et al.,, ). Despite the recent success of standard MARL, its transition from prototypes to reliable production is hindered by robustness concerns due to the complexity and variability of both the real-world environment and human behaviors. Specifically, environmental uncertainty can arise from sim-to-real gaps (Tobin et al.,, ), unexpected disturbance (Pinto et al.,, ), system noise, and adversarial attacks (Mahmood et al.,, ); agents’ behaviors are subject to unknown bounded rationality and variability (Tversky and Kahneman,, ). The solution learned at training time can fail catastrophically when faced with a slightly shifted MARL problem during testing, resulting in a significant drop in overall outcomes and each agent’s individual payoff (Balaji et al.,, ; ; Zeng et al.,, ; Yeh et al.,, ; Shi et al.,, ; Slumbers et al.,, ).

To address robustness challenges, a promising framework is (distributionally) robust Markov games (RMGs) (Littman,, ; Shapley,, ). It is a robust counterpart to the common playground of standard MARL problems — Markov games (MGs) (; Kardeş et al.,, ). In standard MGs, agents consider (competitive) personal objectives and simultaneously interact with each other within a shared unknown environment. The goal is to learn some solution concepts called equilibria, which are joint strategies/policies of agents that all of them stick with rationally with other agents fixed; for instance, Nash equilibria (NE) (Nash,, ; Shapley,, ), correlated equilibria (CE), and coarse correlated equilibria (CCE) (Aumann,, ; Moulin and Vial,, ). To promote robustness, RMGs differ from standard MGs by defining each agent’s payoff (objective) as its worst-case performance when the dynamics of the game shift within a prescribed uncertainty set centered around a nominal environment.

### 1.1 The curse of multiagency in robust MARL

Sample efficiency is a crucial metric for MARL due to the limited availability of data relative to the high dimensionality of the problem. In MARL, agents strive to learn a rationally optimal solution (equilibrium) through interactions with an unknown environment (Silver et al.,, ; Vinyals et al.,, ; Achiam et al.,, ). In contemporary applications, the environment is often extremely large-scale, while data acquisition can be prohibitively limited by high costs and stakes. As such, a notable challenge in terms of scalability for sample efficiency in MARL is known as the curse of multiagency — the sample complexity requirement scales exponentially with the number of agents (induced by the exponentially growing size of the joint action space). This issue has been recognized and studied in extensive MARL problems, but remains open for robust MARL. We concentrate on learning finite-horizon multi-player general-sum Markov games with a generative model (Kearns and Singh,, ), where the number of agents is $n$, the episode length is $H$, the size of the state space is $S$, and the size of the $i$-th agent’s action space is $A_{i}$, for $1\leq i\leq n$.

- •
Breaking the curse of multiagency in standard MARL. A line of pioneering work (Jin et al.,, ; Bai and Jin,, ; Song et al.,, ; Li et al.,, ) has recently introduced a new suite of algorithms using adaptive sampling that provably break the curse of multiagency in standard MGs. In particular,
to find an $\varepsilon$-approximate CCE, Li et al., () requires a sample complexity no more than
$\displaystyle\widetilde{O}\left(\frac{H^{4}S\sum_{i=1}^{n}A_{i}}{\varepsilon^{2}}\right)$
(1)
up to logarithmic factors, which depends only on the sum of individual actions, rather than the number of joint actions.
- •
The persistent curse of multiagency in robust MARL. The development of provable sample-efficient algorithms for RMGs is largely underexplored, with only a few recent studies (; Kardeş et al.,, ; Ma et al.,, ; Blanchet et al.,, ; Shi et al.,, ). Focusing on a class of RMGs with uncertainty sets satisfying the $(s,\bm{a})$-rectangularity condition, existing works all suffer from the curse of multiagency, significantly limiting their scalability. For example, using the total variation (TV) distance as the divergence function, the state-of-the-art (Shi et al.,, ), using non-adaptive sampling, finds an $\varepsilon$-approximate robust CCE with a sample complexity no more than
$\displaystyle\widetilde{O}\left(\frac{H^{3}S\prod_{i=1}^{n}A_{i}}{\varepsilon^{2}}\min\Big{\{}H,~{}\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}\right)$
(2)
up to logarithmic factors, where $\sigma_{i}\in[0,1)$ is the uncertainty level for the $i$-th agent. As a result, the sample size requirement becomes prohibitive when the number of agents is large.

Consequently, there is a significant desire to explore paths that could break through the curse of multiagency in RMGs, which is much more involved than its standard counterpart due to complicated non-linearity introduced by planning for worst-case performances. Nevertheless, the family of RMGs is a much richer class of problems because of the flexibility in choosing the uncertainty sets to capture different robust design considerations. While convenient, the $(s,\bm{a})$-rectangularity condition prevalent in current approaches can be overly restricted in practice, as each agent’s uncertainty set is assumed to be independent of other agents’ strategies and can be decoupled into independent subsets for each state-joint action pair $(s,\bm{a})$, suggesting it might be challenging to break the curse of multiagency in the existing framework. Given these limitations, we are motivated to develop new classes of RMGs that can provide robust solutions applicable to more realistic MARL problems with sample-efficient algorithms. This raises an open question:

Can we design RMGs with practically-meaningful uncertainty sets that come with sample complexity guarantees breaking the curse of multiagency?

**Table 1: Comparisons between our results and prior art for finding an $\varepsilon$-approximate equilibrium in finite-horizon multi-agent general-sum robust Markov games. We omit all logarithmic factors in the sample complexities here. Our result is the only computationally tractable algorithm that provably breaks the curse of multiagency.**
| Algorithm | Uncertainty set | Equilibria | Sample complexity |
| --- | --- | --- | --- |
| P^2MPO $\frac{1^{7}}{1^{7^{7}}}$ | $(s,\bm{a})$-rectangularity | robust NE | $S^{4}\left(\prod_{i=1}^{n}A_{i}\right)^{3}H^{4}/\varepsilon^{2}$ |
| (Blanchet et al.,, ) |  |  |  |
| DR-NVI $\frac{1^{7}}{1^{7^{7}}}$ | $(s,\bm{a})$-rectangularity | robust NE/CE/CCE | $\frac{SH^{3}\prod_{i=1}^{n}A_{i}}{\varepsilon^{2}}\min\Big{\{}H,~{}\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}$ |
| (Shi et al.,, ) |  |  |  |
| Robust-Q-FTRL $\frac{1^{7}}{1^{7^{7}}}$ | fictitious |  |  |
| (this work) | $(s,a_{i})$-rectangularity | robust CCE | $\frac{SH^{6}\sum_{1\leq i\leq n}A_{i}}{\varepsilon^{4}}\min\Big{\{}H,~{}\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}$ |

### 1.2 Contributions

We propose a new class of RMGs with a fictitious uncertainty set that explicitly captures uncertainties in the environment in view of other agents’ strategies, making it suitable for complex real-world scenarios. We begin by verifying the game-theoretic properties of the proposed class of RMGs to ensure the existence of robust variants of well-known standard equilibria notions, robust NE and robust CCE. Next, due to the general intractability of learning NE, we focus on designing algorithms that can provably overcome the curse of multiagency in learning an approximate robust CCE, referring to a joint policy where no agent can improve their benefit by more than $\varepsilon$ through rational deviations.. Specifically, for sampling mechanisms to explore the unknown environment, we assume access to a generative model that can only draw samples from the nominal environment (Shi et al.,, ). The main contributions are summarized as follows.

- •
We introduce a new class of robust Markov games using fictitious uncertainty sets with policy-induced $(s,a_{i})$-rectangularity condition (see Section [2.2](#S2.SS2) for details), which is a natural adaptation from robust single-agent RL to robust MARL. The uncertainty set for each agent $i$ can be decomposed into independent subsets over each state and its own action tuple $(s,a_{i})$, where each subset is a “ball” around the expected nominal transition determined by other agents’ policies and the nominal transition kernel, a distance function $\rho$, and the radius/uncertainty level $\sigma_{i}$.
We verify several essential facts of this class of RMGs: the existence of the desired equilibrium — robust NE and robust CCE for this new class of RMGs using game-theoretical tools such as fixed-point theorem; the existence of best-response policies and robust Bellman equations.
- •
We consider the total variation (TV) distance as the distance metric $\rho$ for uncertainty sets due to its popularity in both theory (Panaganti and Kalathil,, ; Shi et al.,, ; Blanchet et al.,, ; Shi et al.,, ) and practice (Pan et al.,, ; Lee et al.,, ; Szita et al.,, ). We propose Robust-Q-FTRL that can provably find $\varepsilon$-approximate robust CCE with high probability, as long as the sample size exceeds
$\displaystyle\widetilde{O}\left(\frac{SH^{6}\sum_{i=1}^{n}A_{i}}{\varepsilon^{4}}\min\Big{\{}H,~{}\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}\right)$
(3)
up to logarithmic factors, where $\sigma_{i}\in(0,1]$ is the uncertainty level for the $i$-th agent.
To the best of our knowledge, this is the first algorithm to break the curse of multiagency in sample complexity of RMGs. It provably finds an $\varepsilon$-approximate robust CCE using a sample size that is polynomial to all salient parameters. Table [1](#S1.T1) provides a detailed comparison to prior works in robust MARL, where our results significantly improve upon prior art ([2](#S1.E2)) (Shi et al.,, ) by reducing the exponential dependency on the size of each agent’s action space to a linear dependency. To achieve this, we utilize adaptive sampling and online adversarial learning tools, coupled by a tailored design and analysis for robust MARL due to the nonlinearity of the robust value function, which contrasts with the linear payoff functions in standard MARL with respect to the transition kernel.

##### Notation.

In this paper, we denote $[T]\coloneqq\{1,2,\dots,T\}$ for any positive integer $T>0$. We define $\Delta({\mathcal{S}})$ as the simplex over a set ${\mathcal{S}}$. For any policy $\pi$ and function $Q(\cdot)$ defined over a domain $\mathcal{B}$, the variance of $Q$ under $\pi$ is given by $\mathsf{Var}_{\pi}(Q)\coloneqq\sum_{a\in\mathcal{B}}\pi(a)[Q(a)-\mathbb{E}_{\pi}[Q]]^{2}$. We define $x=[x(s,\mathbf{a})]_{(s,\mathbf{a})\in\mathcal{S}\times\mathcal{A}}\in\mathbb{R}^{SA}$ as any vector that represents values for each state-action pair, and $x=[x(s,a_{i})]_{(s,a_{i})\in\mathcal{S}\times\mathcal{A}_{i}}\in\mathbb{R}^{SA_{i}}$ as any vector representing agent-wise state-action values. Similarly, we denote $x=[x(s)]_{s\in\mathcal{S}}$ as any vector representing values for each state. For $\mathcal{X}\coloneqq({\mathcal{S}},\{A_{i}\}_{i\in[n]},H,\{\sigma_{i}\}_{i\in[n]},\frac{1}{\varepsilon},\frac{1}{\delta})$, let $f(\mathcal{X})=O(g(\mathcal{X}))$ denote that there exists a universal constant $C_{1}>0$ such that $f\leq C_{1}g$. Furthermore, the notation $\widetilde{O}(\cdot)$ is defined similarly to ${O}(\cdot)$ but hides logarithmic factors.

### 1.3 Related work

##### Breaking curse of multiagency for standard Markov games.

Breaking the curse of multiagency is a major and prevalent challenge in sequential games.
In standard multi-agent general-sum MGs, it has been shown that learning a Nash equilibrium requires an exponential sample complexity (Song et al.,, ; Rubinstein,, ; Bai and Jin,, ). However, for other types of equilibria, such as CE and CCE, many works have successfully broken the curse of multiagency. Specifically, for finite-horizon general-sum MGs in the tabular setting with finite state and action spaces, Jin et al., () developed the V-learning algorithm for learning CE and CCE with the sample complexity of $\widetilde{O}(H^{6}S(\max_{i\in[n]}A_{i})^{2}/\epsilon^{2})$ and $\widetilde{O}(H^{6}S\max_{i\in[n]}A_{i}/\epsilon^{2})$, respectively; Daskalakis et al., () achieved a sample complexity of $\widetilde{O}(H^{11}S^{3}\max_{i\in[n]}A_{i}/\epsilon^{3})$ for learning a CCE. Beyond tabular settings, Wang et al., () and Cui et al., () extended these results to linear function approximation, achieving sample complexities of $\widetilde{O}(d^{4}H^{6}\left(\max_{i\in[n]}A_{i}^{5}\right)/\epsilon^{2})$ and $\widetilde{O}(H^{10}d^{4}\log\left(\max_{i\in[n]}A_{i}\right)/\epsilon^{4})$, respectively, where $d$ is the dimension of the linear features. For Markov potential games, a subclass of MGs, Song et al., () provided a centralized algorithm that learns a NE with a sample complexity of $\widetilde{O}(H^{4}S^{2}\max_{i\in[n]}A_{i}/\epsilon^{3})$.

##### Finite-sample analysis for distributionally robust Markov games.

Robust Markov games under environmental uncertainty are largely underexplored, with only a few provable algorithms (; Kardeş et al.,, ; Ma et al.,, ; Blanchet et al.,, ; Shi et al.,, ).
Existing sample complexity analyses all suffer from the daunting curse of multiagency issues, or impose an extremely restricted uncertainty level that can fail to deliver the desired robustness (Ma et al.,, ; Blanchet et al.,, ; Shi et al.,, ). Specifically, they all consider a class of RMGs with the $(s,\bm{a})$-rectangularity condition, where the uncertainty sets for each agent can be decomposed into independent sets over each $(s,\bm{a})$ pair. Shi et al., () considered the generative model with an uncertainty set measured by the TV distance, Blanchet et al., () treated a different sampling mechanism with offline data for both the TV distance and KL divergence. In addition, Ma et al., () required the uncertainty level be much smaller than the accuracy-level and an instance-dependent parameter (i.e., $\sigma_{i}\leq\max\{\frac{\varepsilon}{SH^{2}},\frac{p_{\text{min}}}{H}\}$ for all $i\in[n]$). This can thus fail to maintain the desired robustness, especially when the accuracy requirement is high (i.e., $\varepsilon\rightarrow 0$) or the RMG has small minimal positive transition probabilities (i.e., $p_{\text{min}}\rightarrow 0$).

##### Robust MARL.

Standard MARL algorithms may overfit the training environment and could fail dramatically due to the perturbations and variability of both agents’ behaviors and the shared environment, leading to performance drop and large deviation from the equilibrium. To address this, this work considers a robust variant of MARL adopting the distributionally robust optimization (DRO) framework that has primarily been investigated in supervised learning (Rahimian and Mehrotra,, ; Gao,, ; Bertsimas et al.,, ; Duchi and Namkoong,, ; Blanchet and Murthy,, ) and has attracted a lot of attention in promoting robustness in single-agent RL (Nilim and El Ghaoui,, ; Iyengar,, ; Badrinath and Kalathil,, ; Zhou et al.,, ; Shi and Chi,, ; Wang et al.,, ; Shi et al.,, ). Beyond the RMG framework considered in this work, recent research has advanced the robustness of MARL algorithms from various perspectives, including resilience to uncertainties or attacks on states (Han et al.,, ; Zhou and Liu,, ), the type of agents (Zhang et al.,, ), other agents’ policies (Li et al.,, ; Kannan et al.,, ), offline data poisoning (Wu et al.,, ; McMahan et al.,, ), and nonstationary environment (Szita et al.,, ). A recent review can be found in Vial et al., ().

## 2 Preliminaries

In this section, we begin with some background on multi-agent general-sum standard Markov games (MGs) in finite-horizon
settings, followed by a general framework of a robust variant of standard MGs —- distributionally robust Markov games.

### 2.1 Standard Markov games

A finite-horizon multi-agent general-sum Markov game (MG) is characterized by the tuple

$$ $\mathcal{MG}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},P,r,H\big{\}}.$ $$

This setup features $n$ agents each striving to maximize their individual long-term cumulative rewards within a shared environment.
At each time step, all agents observe the same state over the state space ${\mathcal{S}}=\{1,\cdots,S\}$ within the shared environment. For each agent $i$ ($i\in[n]$), $\mathcal{A}_{i}=\{1,\cdots,A_{i}\}$ denotes its action space containing $A_{i}$ possible actions. The joint action space for all agents (resp. the subset excluding the $i$-th agent) is defined as $\mathcal{A}\coloneqq\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{n}$ (resp.  $\mathcal{A}_{-i}\coloneqq\prod_{j\neq i}\mathcal{A}_{j}$ for any $i\in[n]$). We use the notation $\bm{a}\in\mathcal{A}$ (resp. $\bm{a}_{-i}\in\mathcal{A}_{-i}$) to denote a joint action profile involving all agents (resp. all except the $i$-th agent). In addition, the probability transition kernel $P=\{P_{h}\}_{1\leq h\leq H}$, with each $P_{h}:{\mathcal{S}}\times\mathcal{A}\mapsto\Delta({\mathcal{S}})$, describes the dynamics of the game: $P_{h}(s^{\prime}\,|\,s,\bm{a})$ is the probability of transitioning from state $s\in{\mathcal{S}}$ to state $s^{\prime}\in{\mathcal{S}}$ at time step $h$ when agents choose the joint action profile $\bm{a}\in\mathcal{A}$.
The reward function of the game is $r=\{r_{i,h}\}_{1\leq i\leq n,1\leq h\leq H}$, with each $r_{i,h}:{\mathcal{S}}\times\mathcal{A}\mapsto[0,1]$ normalized to the unit interval. For any $(i,h,s,\bm{a})\in[n]\times[H]\times{\mathcal{S}}\times\mathcal{A}$, $r_{i,h}(s,\bm{a})$ represents the immediate reward received by the $i$-th agent in state $s$ when the joint action profile $\bm{a}$ is taken. Last but not least, $H>0$ represents the horizon length.

##### Markov policies and value functions.

In this work, we concentrate on Markov policies that the action selection rule depends only on the current state $s$, independent from previous trajectory. Namely, the $i$-th ($i\in[n]$) agent chooses actions according to $\pi_{i}=\{\pi_{i,h}:{\mathcal{S}}\mapsto\Delta(\mathcal{A}_{i})\}_{1\leq h\leq H}$. Here, $\pi_{i,h}(a\,|\,s)$ represents the probability of selecting action $a\in\mathcal{A}_{i}$ in state $s$ at time step $h$. As such, the joint Markov policy of all agents can be denoted as $\pi=(\pi_{1},\ldots,\pi_{n}):{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$, i.e., given any $s\in{\mathcal{S}}$ and $h\in[H]$, the joint action profile $\bm{a}\in\mathcal{A}$ of all agents is chosen following the distribution $\pi_{h}(\cdot\,|\,s)=(\pi_{1,h},\pi_{2,h}\ldots,\pi_{n,h})(\cdot\,|\,s)\in\Delta(\mathcal{A})$.

To continue, for any given joint policy $\pi$ and transition kernel $P$ of a $\mathcal{MG}$, the $i$-th agent’s long-term cumulative reward can be characterized by the value function $V_{i,h}^{\pi,P}:{\mathcal{S}}\mapsto\mathbb{R}$ (resp. Q-function $Q_{i,h}^{\pi,P}:{\mathcal{S}}\times\mathcal{A}\mapsto\mathbb{R}$) as below: for all $(h,s,a)\in[H]\times{\mathcal{S}}\times\mathcal{A}$,

$$ $\displaystyle V_{i,h}^{\pi,P}(s)$ $\displaystyle\coloneqq\mathbb{E}_{\pi,P}\left[\sum_{t=h}^{H}r_{i,t}\big{(}s_{t},\bm{a}_{t}\big{)}\mid s_{h}=s\right],\quad Q_{i,h}^{\pi,P}(s,\bm{a})\coloneqq\mathbb{E}_{\pi,P}\left[\sum_{t=h}^{H}r_{i,t}\big{(}s_{t},\bm{a}_{t}\big{)}\mid s_{h}=s,\bm{a}_{h}=\bm{a}\right].$ (4) $$

In this context, the expectation is calculated over the trajectory $\{(s_{t},\bm{a}_{t})\}_{h\leq t\leq H}$ produced by following the joint policy $\pi$ under the transition kernel $P$.

### 2.2 Distributionally robust Markov games

A general distributionally robust Markov game (RMG) is represented by the tuple

$$ $\mathcal{RMG}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},\{\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})\}_{1\leq i\leq n},r,H\big{\}}.$ $$

Here, ${\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},r,H$ are defined in the same manner as those in standard MGs (see Section [2.1](#S2.SS1)). RMGs differ from standard MGs: for each agent $i$ ($1\leq i\leq n$), the transition kernel is not fixed but can vary within its own prescribed uncertainty set $\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})$ centered around some
nominal kernel $P^{0}:{\mathcal{S}}\times\mathcal{A}\mapsto\Delta({\mathcal{S}})$ that represents a reference (such as the training environment).
The shape and the size of the uncertainty set $\big{\{}\mathcal{U}^{\sigma_{i}}_{\rho}(P^{0})\big{\}}_{i\in[n]}$ are further specified by a divergence function $\rho$ and the uncertainty levels $\{\sigma_{i}\}_{i\in[n]}$, serving as the “distance” metric and the radius respectively.

Various choices of the divergence function have been considered in the literature of robust RL, including but not limited to $f$-divergence (such as total variation, $\chi^{2}$ divergence, and Kullback-Leibler (KL) divergence) (Yang et al.,, ; Zhou et al.,, ; Shi and Chi,, ; Lu et al.,, ; Wang et al.,, ) and Wasserstein distance (Xu et al.,, ). Adopting uncertainty sets with different structures leads to distinct RMGs, as they address distinct types of uncertainty and game-theoretical solutions. This paper focuses on variability in environmental dynamics (transition kernels), though uncertainty in agents’ reward functions could also be considered similarly but is omitted for brevity.

##### Robust value functions and best-response policies.

For any RMG, each agent seeks to maximize its worst-case performance in the presence of other agents’ behaviors despite perturbations in the environment dynamics, as long as the kernel transitions remain within its prescribed uncertainty set $\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})$. Mathematically, given any joint policy $\pi:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$, the worst-case performance of any agent $i$ is characterized by the robust value function $V_{i,h}^{\pi,\sigma_{i}}$ and the robust Q-function $Q_{i,h}^{\pi,\sigma_{i}}$: for all $(i,h,s,a_{i})\in[n]\times[H]\times{\mathcal{S}}\times\mathcal{A}_{i}$,

$$ $\displaystyle V_{i,h}^{\pi,\sigma_{i}}(s)$ $\displaystyle\coloneqq\inf_{P\in\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})}V_{i,h}^{\pi,P}(s)\qquad\mbox{and}\qquad Q_{i,h}^{\pi,\sigma_{i}}(s,a_{i})\coloneqq\inf_{P\in\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})}Q_{i,h}^{\pi,P}(s,a_{i}).$ (5) $$

Note that different from ([4](#S2.E4)), here the Q-function for any $i$-th agent is defined only over its own action $a_{i}\in\mathcal{A}_{i}$ rather than the joint action $\bm{a}\in\mathcal{A}$.

To continue, we denote $\pi_{-i}$ as the policy for all agents except for the $i$-th agent. By optimizing the $i$-th agent’s policy $\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\rightarrow\Delta(\mathcal{A}_{i})$ (independent from $\pi_{-i}$), we define the maximum of the robust value function as

$$ $\displaystyle V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s)$ $\displaystyle\coloneqq\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}V_{i,h}^{\pi^{\prime}_{i}\times\pi_{-i},\sigma_{i}}(s)=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}\inf_{P\in\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})}V_{i,h}^{\pi_{i}^{\prime}\times\pi_{-i},P}(s)$ (6) $$

for all $(i,h,s)\in[n]\times[H]\times{\mathcal{S}}$.
The policy that achieves the maximum of the robust value function for all $(i,h,s)\in[n]\times[H]\times{\mathcal{S}}$ is called a robust best-response policy.

##### Solution concepts for robust Markov games.

In view of the conflicting objectives between agents, establishing equilibrium becomes the goal of solving RMGs. As such, we introduce two kinds of solution concepts — robust NE and robust CCE — robust variants of standard NE and CCE (usually considered in standard MGs) specified to the form of RMGs.

- •
Robust NE. A product policy $\pi=\pi_{1}\times\pi_{2}\times\cdots\times\pi_{n}:{\mathcal{S}}\times[H]\mapsto\prod_{i=1}^{n}\Delta(\mathcal{A}_{i})$ is
said to be a robust NE if
$V_{i,1}^{\pi,\sigma_{i}}(s)=V_{i,1}^{\star,\pi_{-i},\sigma_{i}}(s),\quad\forall(s,i)\in{\mathcal{S}}\times[n].$
(7)
Given the strategies of the other agents $\pi_{-i}$, when each agent wants to optimize its worst-case performance when the environment and other agents’ policy stay within its own uncertainty set $\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})$, robust NE means that no player can benefit by unilaterally diverging from its present strategy.
- •
Robust CCE.
A distribution over the joint product policy $\xi\coloneqq\{\xi_{h}\}_{h\in[H]}:[H]\mapsto\Delta({\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i}))$ is said to be a robust CCE if it holds that
$\mathbb{E}_{\pi\sim\xi}\left[V_{i,1}^{\pi,\sigma_{i}}(s)\right]\geq\mathbb{E}_{\pi\sim\xi}\left[V_{i,1}^{\star,\pi_{-i},\sigma_{i}}(s)\right],\quad\forall(i,s)\in[n]\times{\mathcal{S}}.$
(8)
Considering all agents follow the policy drawn from the distribution $\xi$, i.e., $\pi_{h}\sim\xi_{h}$ for all $h\in[H]$, when the distribution of all agents but the $i$-th agent’s policy is fixed as the marginal distribution of $\xi$, robust CCE indicates that no agent can benefit from deviating from its current policy.

Note that, for standard MGs, CCE is defined as a possibly correlated joint policy $\pi^{\mathsf{CCE}}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$ (Moulin and Vial,, ; Aumann,, ) if it holds that

$$ $V_{i,1}^{\pi^{\mathsf{CCE}},P}(s)\geq\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\rightarrow\Delta(\mathcal{A}_{i})}V_{i,1}^{\pi^{\prime}_{i}\times\pi^{\mathsf{CCE}}_{-i},P}(s),\qquad\forall(s,i)\in{\mathcal{S}}\times[n].$ (9) $$

This correlated policy $\pi^{\mathsf{CCE}}$ can also be viewed as a distribution $\xi$ over the product policy space since each joint action $\bm{a}$ can be seen as a deterministic product policy.
Careful readers may note that the definition ([9](#S2.E9)) of CCE in standard MGs is in a different form from the one ([8](#S2.E8)) in RMGs, as the latter does not include the expectation operator $\mathbb{E}_{\pi\sim\xi}[\cdot]$ with respect to the policy distribution ($\xi$) over the value function. We emphasize that the definition with the expectation operator outside of the value (or cost) function with respect to a distribution of product pure strategies in ([8](#S2.E8)) is a natural formulation originating from game theory (Moulin et al.,, ; Moulin and Vial,, ). In standard MARL and previous robust MARL studies, the definition in ([9](#S2.E9)) is typically used because ([9](#S2.E9)) and ([8](#S2.E8)) are identical in those situations, as the expectation operator and the corresponding value functions are linear with respect to the joint policy, allowing them to be interchanged (Li et al.,, ; Shi et al.,, ).

## 3 Robust Markov Games with Fictitious Uncertainty Sets

Given the definition of general RMGs, a natural question arises: what kinds of uncertainty sets should we consider to achieve the desired robustness in our solutions? To address this, we focus on a specific class of RMGs characterized by a type of natural yet powerful uncertainty sets.

### 3.1 A novel uncertainty set definition in RMGs

We propose a new class of uncertainty sets, named fictitious uncertainty sets, which count in the uncertainty induced by both the environment and agents’ behaviors in a correlated manner. Before introducing the uncertainty sets, we provide some auxiliary notations as below. We denote a vector of any transition kernel $P:{\mathcal{S}}\times\mathcal{A}\mapsto\Delta({\mathcal{S}})$ or $P^{0}:{\mathcal{S}}\times\mathcal{A}\mapsto\Delta({\mathcal{S}})$ respectively as

$$ $\displaystyle\forall(s,\bm{a})\in{\mathcal{S}}\times\mathcal{A}:\quad$ $\displaystyle P_{h,s,\bm{a}}\coloneqq P_{h}(\cdot\,|\,s,\bm{a})\in\mathbb{R}^{1\times S},\qquad P_{h,s,\bm{a}}^{0}\coloneqq P^{0}_{h}(\cdot\,|\,s,\bm{a})\in\mathbb{R}^{1\times S}.$ (10) $$

For any (possibly correlated) joint Markov policy (defined in [section 2.1](#S2.SS1)) $\pi:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$, we define the expected nominal transition kernel conditioned on the situation that the $i$-th agent chooses some action $a_{i}\in\mathcal{A}_{i}$ and other agents play according to the conditional policy (i.e., $\bm{a}_{-i}\sim\pi_{h}(\cdot\,|\,s,a_{i})$) given $s\in{\mathcal{S}}$ and $a_{i}$ as below: for each time step $h\in[H]$,

$$ $\displaystyle\forall(h,s,a_{i})\in[H]\times{\mathcal{S}}\times\mathcal{A}_{i}:\quad P^{\pi_{-i}}_{h,s,a_{i}}=\mathbb{E}_{\bm{a}\sim\pi_{h}(\cdot\,|\,s,a_{i})}\left[P^{0}_{h,s,\bm{a}}\right]=\sum_{\bm{a}_{-i}\in\mathcal{A}_{-i}}\frac{\pi_{h}(a_{i},\bm{a}_{-i}\,|\,s)}{\pi_{i,h}(a_{i}\,|\,s)}\left[P^{0}_{h,s,\bm{a}}\right].$ (11) $$

Armed with the above definitions, now we are in a position to define the fictitious uncertainty sets, denoted as $\big{\{}\mathcal{U}^{\sigma_{i}}_{\rho}(P^{0},\cdot)\big{\}}_{i\in[n]}$, which satisfy a policy-induced $(s,a_{i})$-rectangularity condition.

###### Definition 1 .

For any joint policy $\pi:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$, divergence function $\rho:\Delta({\mathcal{S}})\times\Delta({\mathcal{S}})\mapsto\mathbb{R}^{+}$ and accessible uncertainty levels $\sigma_{i}\geq 0$ for all $i\in[n]$, the fictitious uncertainty sets $\big{\{}\mathcal{U}^{\sigma_{i}}_{\rho}(P^{0},\pi)\big{\}}_{i\in[n]}$ satisfy the policy-induced $(s,a_{i})$-rectangularity condition:

$$ $\displaystyle\forall i\in[n]:~{}\mathcal{U}^{\sigma_{i}}_{\rho}(P^{0},\pi)$ $\displaystyle\coloneqq\otimes\;\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right),$ $\displaystyle\text{s.t. }\forall(h,s,a_{i})\in[H]\times{\mathcal{S}}\times\mathcal{A}_{i}:\quad\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)$ $\displaystyle\coloneqq\left\{P\in\Delta({\mathcal{S}}):\rho\left(P,P^{\pi_{-i}}_{h,s,a_{i}}\right)\leq\sigma_{i}\right\},$ (12) $$

where $\otimes$ represents the Cartesian product.

In words, conditioned on a fixed joint policy $\pi$, the uncertainty set $\mathcal{U}^{\sigma_{i}}_{\rho}(P^{0},\pi)$ for each $i$-th agent can be decomposed into a Cartesian product of subsets over each state and agent-action pair $(s,a_{i})$. Each uncertainty subset $\mathcal{U}^{\sigma_{i}}_{\rho}(P^{\pi_{-i}}_{h,s,a_{i}})$ over $(s,a_{i})$ is defined as a “ball” around a reference — the expected nominal transition kernel $P^{\pi_{-i}}_{h,s,a_{i}}$ conditioned on both transition kernel and agents’ behavior $\pi$.

###### Definition 1 .

##### Further discussions of fictitious uncertainty sets.

It is in order to remark on the proposed type of uncertainty sets, in comparison with prior works.

- •
A natural adaptation from single-agent robust RL. When agents follow some joint policy $\pi:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$, fixing other agents’ policy $\pi_{-i}$, from the perspective of each individual agent $i$, RMGs with our policy-induced $(s,a_{i})$-rectangularity condition will degrade to a single-agent robust RL problem with the widely used $(s,a_{i})$-rectangularity condition in the single-agent literature (Iyengar,, ; Zhou et al.,, ). Namely, from any agent $i$’s viewpoint, in a RMG, it has an "overall environment" player that can not only manipulate the environmental dynamics but also other players’ policy $\pi_{-i}$.
- •
Allowing uncertainty from both the environment and agents’ behaviors in a correlated manner. One essential feature of our proposed uncertainty set is that it is shaped by both the environment and agents’ strategies in a (possibly) correlated manner. Specifically, for any agent $i$ and a given policy $\pi$, any uncertainty subset $\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)$ (over any $(h,s,a_{i})$) is constructed as a neighborhood around a nominal center $P_{h,s,a_{i}}^{\pi_{-i}}$ (see ([11](#S3.E11))) that depends on both the nominal environment $P^{0}$ and other agents’ conditional strategies $\pi_{h}(\cdot\,|\,s,a_{i})$.
- •
Comparisons to prior works.
Prior works on provable sample-efficient algorithms have focused on a different type of uncertainty sets with $(s,\bm{a})$-rectangularity condition (Ma et al.,, ; Blanchet et al.,, ; Shi et al.,, ). This class of uncertainty sets decouples the uncertainty into independent subsets for each state-joint action pair $(s,\bm{a})$, accounting for the uncertainty of the environment and agents’ strategies independently. In comparison, the proposed uncertainty set lifts this independence assumption across subsets over different $(s,a_{i},\bm{a}_{-i})$ for any $\bm{a}_{-i}\in\mathcal{A}_{i}$, enabling the environment and agents’ strategies to shape the uncertainty set in a correlated manner.

### 3.2 Properties of RMGs with fictitious rectangular uncertainty set

Throughout the paper, we focus on the class of RMGs with the above proposed fictitious uncertainty sets, represented as

$$ $\mathcal{RMG}_{{\pi}}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},\{\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0},\cdot)\}_{1\leq i\leq n},r,H\big{\}}$ $$

and abbreviated as fictitious RMGs in the remaining of the paper. In this section, we present key facts about fictitious RMGs related to best-response policies, equilibria, and the corresponding one-step lookahead robust Bellman equations. The proofs are postponed to Appendix [B](#A2).

First, we introduce the following lemma, which verifies the existence of a robust best-response policy that achieves the maximum robust value function (cf. ([6](#S2.E6))).

###### Lemma 1 .

For any $i\in[n]$, given $\pi_{-i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})$, there exists at least one policy $\widetilde{\pi}_{i}:{\mathcal{S}}\times[H]\rightarrow\Delta(\mathcal{A}_{i})$ for the $i$-th agent that can simultaneously attain $V_{i,h}^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s)$ for all $s\in{\mathcal{S}}$ and $h\in[H]$. We refer this policy as the robust best-response policy.

###### Lemma 1 .

##### Existence of robust NE and robust CCE.

fictitious RMGs can be viewed as hierarchical games with $n+nS\sum_{i=1}^{n}A_{i}$ agents. This includes the original $n$ agents and $n$ additional sets of $S\sum_{i=1}^{n}A_{i}$ independent adversaries, each determining the worst-case transitions for one agent over a state plus agent-wise-action pair.
Considering the solution concepts — robust NE and robust CCE — introduced in Section [2.2](#S2.SS2), the following theorem verifies the existence of them for any fictitious RMGs using Kakutani’s fixed-point theorem (Kakutani,, ), focusing on robust NE firstly.

###### Theorem 1 (Existence of robust NE) .

For any $\mathcal{RMG}_{\pi}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},\{\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0},\cdot)\}_{1\leq i\leq n},$ $r,H\big{\}}$ with an uncertainty set defined in Definition [1](#Thmdefinition1), there exists at least one robust NE.

Analogous to standard Markov games, since $\{\text{robust NE}\}\subseteq\{\text{robust CCE}\}$, Theorem [1](#Thmtheorem1) indicates the existence of robust CCEs directly.

###### Theorem 1 (Existence of robust NE) .

##### Robust Bellman equations.

Fortunately, the class of fictitious RMGs feature a robust counterpart of the Bellman equation — robust Bellman equation.
Specifically, for any joint policy $\pi:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$, the robust value function can be expressed as

$$ $\displaystyle V^{\pi,\sigma_{i}}_{i,h}(s)$ $\displaystyle=\inf_{\mathcal{U}_{\rho}^{\sigma_{i}}\left(P^{0},\pi\right)}\mathbb{E}\left[\sum_{t=h}^{H}r_{i}(s_{t},a_{t})\,|\,s_{h}=s\right]=\mathbb{E}_{\bm{a}\sim\pi_{h}(s)}[r_{i,h}(s,\bm{a})]+\mathbb{E}_{a_{i}\sim\pi_{i,h}(s)}\bigg{[}\inf_{\mathcal{U}_{\rho}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\pi,\sigma_{i}}_{i,h+1}\bigg{]}.$ (13) $$

It can be verified directly by definition.
The robust Bellman equation described above is intrinsically linked to the policy-induced $(s,a_{i})$-rectangularity condition (cf. ([12](#S3.E12))) of the uncertainty set. This condition leads to a well-posed and computationally-tractable class of RMGs by allowing the decomposition from an overall uncertainty set to independent subsets across different agents, time steps, and each state-action pair $(s,a_{i})$.

Note that the specified robust Bellman equation is different for a joint correlated policy and a joint product policy, induced by different expected nominal transition kernels.
In particular, for any joint product policy $\pi:{\mathcal{S}}\times[H]\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$, the expected nominal transition kernel conditioned on the $i$-th agent’s action $a_{i}\in\mathcal{A}_{i}$, current state $s\in{\mathcal{S}}$, and the policy $\pi$ can be expressed by

$$ $\displaystyle P^{\pi_{-i}}_{h,s,a_{i}}$ $\displaystyle=\mathbb{E}_{\bm{a}\sim\pi_{h}(\cdot\,|\,s,a_{i})}\left[P^{0}_{h,s,\bm{a}}\right]=\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,h}(\cdot\,|\,s)}\left[P^{0}_{h,s,(a_{i},\bm{a}_{-i})}\right]$ (14) $$

for any $(i,h,s,a_{i})\in[n]\times[H]\times{\mathcal{S}}\times\mathcal{A}_{i}$,
where the last equality holds since the policy $\pi$ is a product policy, and the distribution of $\bm{a}_{-i}$ is independent of $a_{i}$. It is observed that the expected nominal transition kernel $P^{\pi_{-i}}_{h,s,a_{i}}$ for a product policy $\pi$ is independent of the $i$-th agent’s policy given $(s,a_{i})$. This differs from ([11](#S3.E11)) for a possibly correlated policy, where ([11](#S3.E11)) can generally depend on the $i$-th agent’s policy.

The robust Bellman equation described above is intrinsically linked to the policy-induced $(s,a_{i})$-rectangularity condition (cf. ([12](#S3.E12))) of the uncertainty set. This condition leads to a well-posed and computationally-tractable class of RMGs by allowing the decomposition from an overall uncertainty set to independent subsets across different agents, time steps, and each state-action pair $(s,a_{i})$.

## 4 Sample-Efficient Learning: Algorithm and Theory

In this section, we focus on designing sample-efficient algorithms for solving fictitious RMGs when agents need to collect data by interacting with the unknown shared environment in order to learn the equilibria. To proceed, we shall first specify the data collection mechanism and the divergence function for the uncertainty set. Then we propose a sample-efficient algorithm Robust-Q-FTRL that leverages a carefully-designed adaptive sampling strategy to break the curse of multiagency.

### 4.1 Problem setting and goal

Recall that the uncertainty sets are constructed by specifying a divergence function $\rho$ and the uncertainty level to control its shape and size.
In this work, we focus on using the TV distance as the divergence function $\rho$ for the uncertainty set, following Szita et al., (); Lee et al., (); Pan et al., (); Shi et al., (, ), defined by

$$ $\displaystyle\forall P,P^{\prime}\in\Delta({\mathcal{S}}):\quad\rho_{\mathsf{TV}}\left(P,P^{\prime}\right)\coloneqq\frac{1}{2}\left\|P-P^{\prime}\right\|_{1}.$ (15) $$

For convenience, throughout the paper, we abbreviate $\mathcal{U}^{\sigma_{i}}(\cdot)\coloneqq\mathcal{U}^{\sigma_{i}}_{\rho_{\mathsf{TV}}}(\cdot)$ when there is no ambiguity.

##### Data collection mechanism: a generative model.

We assume the agents interact with the environment through a generative model (simulator) (Kearns and Singh,, ), which is a widely used sampling mechanism in both single-agent RL and MARL (; Li et al.,, ). Specifically, at any time step $h$, we can collect an arbitrary number of independent samples from any state and joint action tuple $(s,\bm{a})\in{\mathcal{S}}\times\mathcal{A}$, generated based on the true nominal transition kernel $P^{0}$:

$$ $\displaystyle s_{h,s,\bm{a}}^{i}\overset{i.i.d}{\sim}P^{0}_{h}(\cdot\,|\,s,\bm{a}),\qquad i=1,2,\ldots$ (16) $$

##### Goal.

Consider any fictitious RMGs $\mathcal{RMG}_{\pi}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},\{\mathcal{U}^{\sigma_{i}}(P^{0})\}_{1\leq i\leq n},r,H\big{\}}$. While learning exact robust equilibria is computationally challenging and may not be necessary in practice, instead
in this work, we focus on finding an approximate robust CCE (defined in ([8](#S2.E8))). Namely, a distribution $\xi\coloneqq\{\xi_{h}\}_{h\in[H]}:[H]\mapsto\Delta({\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i}))$ is said to be an $\varepsilon$-robust CCE if

$$ $\mathsf{gap}_{\mathsf{CCE}}(\xi)\coloneqq\max_{s\in{\mathcal{S}},1\leq i\leq n}\left\{\mathbb{E}_{\pi\sim\xi}\left[V_{i,1}^{\star,\pi_{-i},\sigma_{i}}(s)\right]-\mathbb{E}_{\pi\sim\xi}\left[V_{i,1}^{\pi,\sigma_{i}}(s)\right]\right\}\leq\varepsilon.$ (17) $$

Armed with a generative model of the nominal environment, the goal is to learn a robust CCE using as few samples as possible.

### 4.2 Algorithm design

With the sampling mechanism over a generative model in hand, we propose an algorithm called Robust-Q-FTRL to learn an $\varepsilon$-robust CCE in a sample-efficient manner, summarized in Algorithm [2](#alg2) in the appendix. Robust-Q-FTRL draws inspiration from Q-FTRL developed in the standard MG literature (Li et al.,, ), but empowers tailored designs for learning in fictitious RMGs to achieve a robust equilibrium and to tackle statistical challenges arising from agents’ nonlinear objectives.
Overall, Robust-Q-FTRL takes a single pass to learn recursively from the final time step $h=H$ to $h=1$. At each time step $h\in[H]$, an online learning process with $K$ iterations will be executed. Before introducing the algorithm, we first concentrate on two essential steps customized for learning in fictitious RMGs.

Figure: Algorithm 1 $N$-sample estimation$\big{(}\pi_{h}=\{\pi_{j,h}\}_{j\in[n]},i,h\big{)}$.

##### Constructing the empirical model via N 𝑁 N -sample estimation.

For each time step $h$, we denote $\pi_{i,h}^{k}$ as the current learning policy of the $i$-th agent before the beginning of the $k$-th iteration for any $k\in[K]$. And we denote the joint product policy as $\pi_{h}^{k}=(\pi_{1,h}^{k},\cdots,\pi_{n,h}^{k})$. During each iteration $k$, for each agent $i\in[n]$, we require to generate $N$ independent samples from the generative model over each $(s,a_{i})\in{\mathcal{S}}\times\mathcal{A}_{i}$ to obtain an empirical model, detailed in Algorithm [1](#alg1). It includes an empirical reward function represented by $r_{i,h}^{k}\in\mathbb{R}^{SA_{i}}$ and transition kernels denoted by $P_{i,h}^{k}\in\mathbb{R}^{SA_{i}\times S}$. Note that different from standard MGs, we need to generate $N$ samples instead of $1$ sample per iteration to handle the additional statistical challenges induced by the non-linear objective of agents ($N$ will be specified momentarily).

##### Estimating robust Q-function of the current policy π h k superscript subscript 𝜋 ℎ 𝑘 \pi_{h}^{k} .

We denote $\widehat{V}_{i,h}\in\mathbb{R}^{S}$ as the estimation of the $i$-th agent’s robust value function at time step $h$.
For any agent $i$, with the empirical reward function $r_{i,h}^{k}$, empirical kernel $P_{i,h}^{k}$, and the estimated robust value function $\widehat{V}_{i,h+1}$ at the next step in hand, the robust Q-function $\{q^{k}_{i,h}\}$ of current policy $\pi_{h}^{k}$ can be estimated as:

$$ $\displaystyle\forall(i,h,s,a_{i})\in[n]\times[H]\times{\mathcal{S}}\times A_{i}:\quad q^{k}_{i,h}(s,a_{i})=r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}.$ (20) $$

Unlike the linear function w.r.t. $P_{i,h}^{k}$ in standard MGs, ([20](#S4.E20)) lacks a closed form and introduces an additional inner optimization problem. Solving ([20](#S4.E20)) directly is computationally challenging due to the need to optimize over an $S$-dimensional probability simplex, with complexity growing exponentially with the state space size $S$. Fortunately, by applying strong duality, we can solve ([20](#S4.E20)) equivalently via its dual problem with tractable computation (Iyengar,, ):

$$ $\displaystyle q^{k}_{i,h}(s,a_{i})=r_{i,h}^{k}(s,a_{i})+\max_{\alpha\in[\min_{s}\widehat{V}_{i,h+1}(s),\max_{s}\widehat{V}_{i,h+1}(s)]}\Big{\{}P_{i,h}^{k}\left[\widehat{V}_{i,h+1}\right]_{\alpha}-\sigma_{i}\left(\alpha-\min_{s^{\prime}}\left[\widehat{V}_{i,h+1}\right]_{\alpha}(s^{\prime})\right)\Big{\}},$ (21) $$

where $[V]_{\alpha}$ denotes the clipped version of any vector $V\in\mathbb{R}^{S}$ determined by some level $\alpha\geq 0$, namely,

$$ $\displaystyle[V]_{\alpha}(s)\coloneqq\begin{cases}\alpha,&\text{if }V(s)>\alpha,\\ V(s),&\text{otherwise.}\end{cases}$ (22) $$

The above modules are key components of Robust-Q-FTRL, serving for constructing nonlinear robust objectives in the online learning process and ensuring the desired statistical accuracy.

##### Overall pipeline of Robust-Q-FTRL .

With these technical modules in place, we introduce Robust-Q-FTRL, which follows a similar online learning procedure as Q-FTRL for standard MGs (Li et al.,, ). The complete procedure is summarized in Algorithm [2](#alg2). We denote $Q_{i,h}^{k}\in\mathbb{R}^{SA_{i}}$ as the estimated robust Q-function of the equilibrium for the $i$-th agent at the $k$-th iteration of time step $h$. To begin with, Robust-Q-FTRL initialize the robust value function, robust Q-function $\widehat{V}_{i,H+1}(s)=Q_{i,h}^{0}(s,a_{i})=0$, and the policy $\pi_{i,h}^{1}(a_{i}\,|\,s)=1/A_{i}$ for all $i\in[n]$.
Then subsequently from the final time step $h=H$ to $h=1$, for each step $h$, a $K$ iterations online learning process will be executed. At each $k$-th iteration, given current policy $\pi_{h}^{k}$, as described above, an empirical model ($\{r_{i,h}^{k}\}_{i\in[n]}$ and $\{P_{i,h}^{k}\}_{i\in[n]}$) is constructed by $N$-sample estimation (cf. [algorithm 1](#alg1)). Then the robust Q-function $\{q_{i,h}^{k}\}_{i\in[n]}$ of the current policy $\pi_{h}^{k}$ is estimated by ([21](#S4.E21)).

Now we are ready to specify the loss objective and proceed the online learning procedure.
With the current one-step update $\{q_{i,h}^{k}\}$, we update the Q-estimate as $Q_{i,h}^{k}=(1-\alpha_{k})Q_{i,h}^{k-1}+\alpha_{k}q_{i,h}^{k}$. Here, $\{\alpha_{k}\}_{k\in[K]}$ is a series of rescaled linear learning rates with some $c_{\alpha}\geq 24$,

$$ $\displaystyle\forall k\in[K]:\quad\alpha_{k}=\frac{c_{\alpha}\log K}{k-1+c_{\alpha}\log K}\quad\text{ and }\quad\alpha_{k}^{n}=\begin{cases}\alpha_{k}\prod_{i=k+1}^{n}(1-\alpha_{i}),&\text{if }0<k<n\leq K\\ \alpha_{n}&\text{if }k=n\end{cases}.$ (23) $$

Let the Q-estimate be the online learning loss objective at this moment, we apply the Follow-the-Regularized-Leader strategy (Shalev-Shwartz,, ; Li et al.,, ) to update the corresponding policy as below:

$$ $\displaystyle\pi_{i,h}^{k+1}(a_{i}\,|\,s)=\frac{\exp\big{(}\eta_{k+1}Q_{i,h}^{k}(s,a_{i})\big{)}}{\sum_{a^{\prime}}\exp\big{(}\eta_{k+1}Q_{i,h}^{k}(s,a^{\prime})\big{)}}\quad\text{ with }\quad\eta_{k+1}=\sqrt{\frac{\log K}{\alpha_{k}H}},\qquad k=1,2,\ldots$ $$

This is a widely used adaptive sampling and learning procedure for MARL problems.

After completing $K$ iterations for time step $h$, we finalize the robust value function estimation by setting it to its confidence upper bound, incorporating carefully designed optimistic bonus terms $\{\beta_{i,h}\}$ as: for all $(i,h,s)\in[n]\times[H]\times{\mathcal{S}}$,

$$ $\displaystyle\beta_{i,h}(s)=c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{k=1}^{K}\alpha_{k}^{K}\left\{\mathsf{Var}_{\pi_{i,h}^{k}(\cdot\mid s)}\left(q_{i,h}^{k}(s,\cdot)\right)+H\right\},$ (24) $$

where $c_{\mathsf{b}}$ denotes some absolute constant, $\delta\in(0,1)$ is the high probability threshold,
Finally, after the recursive learning process ends for all time steps $h=H,H-1,\cdots,1$, we output a distribution of product policy $\widehat{\xi}=\{\widehat{\xi}_{h}\}_{h\in[H]}$ over all the policies $\{\pi_{h}^{k}=(\pi_{1,h}^{k}\times\cdots\times\pi_{n,h}^{k})\}_{h\in[H],k\in[K]}$ occurs during the process that defined as

$$ $\displaystyle\forall(h,k)\in[H]\times[K]:\quad\xi_{h}(\pi_{h}^{k})\coloneqq\alpha_{k}.$ (25) $$

Figure: Algorithm 2 Robust-Q-FTRL

### 4.3 Theoretical guarantees

In this section, we provide the theoretical guarantees for the sample complexity of our proposed algorithm Robust-Q-FTRL, shown as below:

###### Theorem 2 (Upper bound) .

Using the TV uncertainty set defined in ([15](#S4.E15)). Consider any $\delta\in(0,1)$ and any fictitious RMGs $\mathcal{RMG}_{\pi}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},\{\mathcal{U}^{\sigma_{i}}(P^{0},\cdot)\}_{1\leq i\leq n},r,H\big{\}}$ with $\sigma_{i}\in(0,1]$ for all $i\in[n]$. For any $\varepsilon\leq\sqrt{\min\big{\{}H,\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\big{\}}}$, Algorithm [2](#alg2) can output an $\varepsilon$-robust CCE $\widehat{\xi}$, i.e.,

$$ $\displaystyle\mathsf{gap}_{\mathsf{CCE}}(\widehat{\xi})\coloneqq\max_{s\in{\mathcal{S}},1\leq i\leq n}\left\{\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\star,\pi_{-i},\sigma_{i}}(s)\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\pi,\sigma_{i}}(s)\right]\right\}\leq\varepsilon$ $$

with probability at least $1-\delta$, as long as

$$ $\displaystyle N\geq\frac{C_{1}H^{2}}{\epsilon^{2}}\min\left\{\frac{1}{\min_{1\leq i\leq n}\sigma_{i}},H\right\},\quad K\geq\frac{C_{1}H^{3}}{\epsilon^{2}}.$ (27) $$

Here $C_{1}$ is some universal large enough constant.
Namely, it is sufficient if the total number of samples acquired in the learning process obeys

$$ $\displaystyle N_{\mathsf{all}}\coloneqq HKNS\sum_{1\leq i\leq n}A_{i}\geq\frac{(C_{1})^{2}H^{6}S\sum_{1\leq i\leq n}A_{i}}{\varepsilon^{4}}\min\Big{\{}H,\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}.$ $$

Before we jump into more discussions of the above theorem, in addition, we introduce the information-theoretic minimax lower bound for this problem as well.

###### Theorem 2 (Upper bound) .

##### Lower bound for learning in fictitious RMGs.

Considering the instances of fictitious RMGs that the action space for all the agents except the $i$-th agent contains only a single action, i.e., $A_{j}=1$ for all $j\neq i$. As such, all the agents $j\neq i$ will take a fixed action and the game reduces to a single-agent robust MDP with $(s,a)$-rectangularity condition (Zhou et al.,, ). So the goal of finding the robust equilibrium — robust NE/CCE also degrades to finding the optimal policy of the $i$-th agent. Invoking the results from Shi et al., (, Theorem 2), the lower bound for the class of fictitious RMGs is achieved directly:
consider any tuple $\big{\{}S,\{A_{i}\}_{1\leq i\leq n},\{\sigma_{i}\}_{1\leq i\leq n},H\big{\}}$ obeying $\sigma_{i}\in(0,1-c_{0}]$ with $0<c_{0}\leq\frac{1}{4}$ being any small enough positive constant, and $H>16\log 2$. Let

$$ $\displaystyle\varepsilon\leq\begin{cases}\frac{c_{1}}{H},&\text{if }\sigma_{i}\leq\frac{c_{1}}{2H},\\ 1&\text{otherwise}\end{cases}$ (28) $$

for any $c_{1}\leq\frac{1}{4}$.
We can construct a set of fictitious RMGs $\mathcal{M}=\{\mathcal{RMG}_{i}\}_{i\in[I]}$, such that for any dataset generated from the nominal environment with in total $N_{\mathsf{all}}$ independent samples over all state-action pairs, we have

$$ $\displaystyle\inf_{\widehat{\xi}\in[H]\mapsto\Delta({\mathcal{S}}\mapsto\prod_{i=1}^{n}\mathcal{A}_{i})}\;\max_{\mathcal{MG}_{i}\in\mathcal{M}}\left\{\mathbb{P}_{\mathcal{MG}_{i}}\big{(}\mathsf{gap}_{\mathsf{CCE}}(\widehat{\xi})>\varepsilon\big{)}\right\}$ $\displaystyle\geq\frac{1}{8},$ (29) $$

provided that

$$ $\displaystyle N_{\mathsf{all}}\leq\frac{C_{2}SH^{3}\max_{1\leq i\leq n}A_{i}}{\varepsilon^{2}}\min\Big{\{}H,\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}.$ (30) $$

Here, the infimum is taken over all estimators $\widehat{\xi}$, $\mathbb{P}_{\mathcal{RMG}_{i}}$ denotes the probability when the game is $\mathcal{MG}_{i}$ for all $\mathcal{MG}_{i}\in\mathcal{M}$, and $C_{2}$ is some small enough constant.

Armed with both the upper bound (Theorem [2](#Thmtheorem2)) and lower bound in ([30](#S4.E30)), we are now ready to discuss the implications of our sample complexity results.

##### Breaking the curse of multiagency in the sample complexity for RMGs.

Theorem [2](#Thmtheorem2) demonstrates that for any fictitious RMGs, Robust-Q-FTRL algorithm finds an $\epsilon$-robust CCE when the total number of samples exceeds

$$ $\displaystyle\widetilde{O}\left(\frac{SH^{6}\sum_{1\leq i\leq n}A_{i}}{\epsilon^{4}}\min\left\{H,\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\right\}\right).$ (31) $$

To the best of our knowledge, Robust-Q-FTRL with the above sample complexity in ([31](#S4.E31)) is the first algorithm for RMGs breaking the curse of multiagency, regardless of the types of uncertainty sets. Our sample complexity depends linearly on the sum of each agent’s actions $\sum_{i=1}^{n}A_{i}$ rather than their product $\prod_{i=1}^{n}A_{i}$—making the algorithm highly scalable as the number of agents increases. Nonetheless, there still exist gaps between our upper bound and the lower bound—especially in terms o the dependency on the horizon length $H$ and the accuracy level $\varepsilon$—an interesting direction to investigate in the future.

##### Comparisons with prior works.

All prior works focus on learning equilibria for a different kind of robust MGs with $(s,\bm{a})$-rectangular uncertainty sets (Ma et al.,, ; Blanchet et al.,, ; Shi et al.,, ). However, the state-of-the-art sample complexity $\widetilde{O}\left(\frac{SH^{3}\prod_{i=1}^{n}A_{i}}{\varepsilon^{2}}\min\Big{\{}H,~{}\frac{1}{\min_{1\leq i\leq n}\sigma_{i}}\Big{\}}\right)$ (Shi et al.,, ) still suffers from the curse of multiagency with an exponential dependency on the number of agents when all agents have equal action spaces, which uses nonadaptive sampling. Our work circumvents the curse of multiagency by resorting to a tailored adaptive sampling and online learning procedure, together with the introduction of a new class of fictitious RMGs, providing a fresh perspective to learning RMGs.

##### Technical insights.

For sample complexity analysis, while previous works have addressed the curse of multiagency in sequential games like standard Markov games (MGs) and Markov potential games, these methods are not directly applicable to RMGs. Prior approaches assume a linear relationship between the value function and the transition kernel, allowing statistical errors across $K$ iterations to cancel out. However, in RMGs, the robust value function, due to its distributionally robust requirement, is highly nonlinear and often lacks a closed form, making it impossible to linearly aggregate statistical errors. To tackle the nonlinear challenges in RMGs, we design a variance-style bonus term through non-trivial decomposition and control of auxiliary statistical errors caused by nonlinearity, resulting in a tight upper bound on regret during the online learning process.

## 5 Conclusion

Robustness in MARL presents greater challenges than in single-agent RL due to the strategic interactions between agents in a game-theoretic setting. This work proposes a new class of RMGs with fictitious uncertainty sets that naturally extends from robust single-agent RL and addresses more realistic scenarios where each agent’s uncertainty is influenced by both the environment and the behavior of others. We then propose Robust-Q-FTRL, the first algorithm to break the curse of multiagency in robust Markov games regardless of the uncertainty set definitions, with sample complexity scaling polynomially with all key parameters. This opens up new research directions in MARL, such as uncertainty set selection, equilibrium refinement, and sample-efficient algorithm design.

## Acknowledgements

The work of Y. Chi is supported in part by the grants NSF CCF-2106778 and CNS-2148212, and by funds from federal agency and industry partners as specified in the Resilient & Intelligent NextG Systems (RINGS) program. The work of L. Shi is supported in part by the Resnick Institute and Computing, Data, and Society Postdoctoral Fellowship at California Institute of Technology. The work of E. Mazumdar is supported in part from NSF-2240110. The work of A. Wierman is supported in part from
the NSF through CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.

## Appendix A Preliminaries

Denoting the vectors $x=[x_{i}]_{1\leq i\leq n}$ and $y=[y_{i}]_{1\leq i\leq n}$, we use the notation ${x}\leq{y}$ (or ${x}\geq{y}$) to signify that $x_{i}\leq y_{i}$ (or $x_{i}\geq y_{i}$) for every $1\leq i\leq n$.
The Hadamard product of two vectors $x$ and $y$ in $\mathbb{R}^{S}$ is denoted as $x\circ y=\big{[}x(s)\cdot y(s)\big{]}_{s\in\mathcal{S}}$.
In addition, for any series of vectors $\{x_{i}\}_{i\in[S}$, $\mathrm{diag}(x_{1},x_{2},\cdots,x_{S})$ denote a block diagonal matrix by placing each given vector $x_{i}$ along the diagonal, with zeros filling the off-diagonal blocks.${0}$ (or ${1}$) represents the all-zero (or all-one) vector, while $e_{i}\in\mathbb{R}^{S}$ denotes a basis vector of dimension $S$ with 1 in the $i$-th position and 0 elsewhere.

### A.1 Additional matrix and vector notation

Before continuing, we introduce or recall some matrix and vector notation that will be used throughout the paper. In particular, for any joint policy $\pi:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A})$ and any $(i,h)\in[n]\times[H]$:

##### Matrices for policy.

We introduce three matrices associated with $\pi$, i.e., $\Pi_{h}^{\pi}\in\mathbb{R}^{S\times S\prod_{i=1}^{n}A_{i}}$, $\Pi_{h}^{\pi_{-i}}\in\mathbb{R}^{S\times S\prod_{j\neq i}A_{j}}$, and $\Pi_{h}^{\pi_{i}}\in\mathbb{R}^{S\times SA_{i}}$, which are defined as block diagonal matrices that adhere to the following properties:

- •
The matrix $\Pi_{h}^{\pi}$ is given by $\mathrm{diag}\left(\pi_{h}(1)^{\top},\pi_{h}(2)^{\top},\ldots,\pi_{h}^{\top}(S)\right)$, where $\pi_{h}(s)=\left[\pi_{h}(\mathbf{a}\,|\,s)\right]_{\mathbf{a}\in\mathcal{A}}\in\Delta(\mathcal{A})$ for each $s\in\mathcal{S}$ represents the joint policy vectors across all agents.
- •
The matrix $\Pi_{h}^{\pi_{-i}}$ can be expressed as $\mathrm{diag}\left(\pi_{-i,h}(1)^{\top},\pi_{-i,h}(2)^{\top},\ldots,\pi_{-i,h}^{\top}(S)\right)$, where $\pi_{-i,h}(s)=\left[\pi_{h}(\bm{a}_{-i}\,|\,s)\right]_{\bm{a}_{-i}\in\mathcal{A}_{-i}}\in\Delta(\mathcal{A}_{-i})$ for all $s\in\mathcal{S}$ denotes the joint policy vectors from all agents except agent $i$.
- •
The matrix $\Pi_{h}^{\pi_{i}}$ is defined as $\mathrm{diag}\left(\pi_{i,h}(1)^{\top},\pi_{i,h}(2)^{\top},\ldots,\pi_{i,h}^{\top}(S)\right)$, where $\pi_{i,h}(s)=\left[\pi_{i,h}(a_{i}\,|\,s)\right]_{a_{i}\in\mathcal{A}_{i}}\in\Delta(\mathcal{A}_{i})$ for each $s\in\mathcal{S}$ represents the policy of the $i$-th agent.

##### Reward vectors.

We recall the definition of $r_{i,h}$ and introduce the reward vectors $r_{i,h}^{\pi}$ and $r_{i,h}^{\pi_{-i}}$ as follows:

- •
Let $r_{i,h}=[r_{i,h}(s,\mathbf{a})]_{(s,\mathbf{a})\in\mathcal{S}\times\mathcal{A}}\in\mathbb{R}^{S\prod_{i=1}^{n}A_{i}}$ represent the reward function for the $i$-th player at time step $h$, where $\mathcal{S}$ is the state space and $\mathcal{A}$ is the action space.
- •
The reward vector $r_{i,h}^{\pi}\in\mathbb{R}^{S}$ corresponds to the joint policy $\pi=\{\pi_{h}\}_{h\in[H]}$ at time step $h$. Specifically, for each $s\in\mathcal{S}$, $r_{i,h}^{\pi}(s)=\mathbb{E}_{\mathbf{a}\sim\pi_{h}(s)}[r_{i,h}(s,\mathbf{a})]$, where the expectation is taken over the actions $\mathbf{a}$ drawn from policy $\pi_{h}$ in state $s$.
- •
The reward vector $r_{i,h}^{\pi_{-i}}\in\mathbb{R}^{SA_{i}}$ corresponds to the joint policy $\pi_{-i}=\{\pi_{-i,h}\}_{h\in[H]}$ at time step $h$, excluding agent $i$. Specifically, for all $s\in\mathcal{S}$ and $a_{i}\in A_{i}$, $r_{i,h}^{\pi_{-i}}(s,a_{i})=\mathbb{E}_{\mathbf{a}_{-i}\sim\pi_{-i,h}(s)}[r_{i,h}(s,\mathbf{a})]$, where the expectation is over the actions $\mathbf{a}_{-i}$ drawn from the joint policy $\pi_{-i,h}$ for all agents except agent $i$.

##### Matrices for transition variants.

We first introduce the following notations related to transitions associated with the nominal transition kernel and the policy $\pi$:

- •
Define $P_{h}^{0}\in\mathbb{R}^{S\prod_{i=1}^{n}A_{i}\times S}$, the matrix representing the nominal transition kernel at time step $h$. Specifically, for any $(s,\mathbf{a})\in\mathcal{S}\times\mathcal{A}$, $P_{h,s,\mathbf{a}}^{0}\in\mathbb{R}^{1\times S}$ represents the row corresponding to the state-action pair $(s,\mathbf{a})$.
- •
Define $P_{h}^{\pi_{-i}}\in\mathbb{R}^{SA_{i}\times S}$, the matrix representing the nominal transition kernel at time step $h$, associated with the joint policy $\pi_{-i}$. Specifically, for all $s,s^{\prime}\in\mathcal{S}$ and $a_{i}\in\mathcal{A}_{i}$,
$P_{h,s,a_{i}}^{\pi_{-i}}(s^{\prime})=\mathbb{E}_{\mathbf{a}_{-i}\sim\pi_{-i,h}(s)}[P_{h,s,\mathbf{a}}^{0}(s^{\prime})].$
Here, $P_{h,s,a_{i}}^{\pi_{-i}}\in\mathbb{R}^{1\times S}$ represents the row corresponding to the state-action pair $(s,a_{i})$.
- •
Let $\widehat{P}_{i,h}^{\pi_{-i}}\in\mathbb{R}^{SA_{i}\times S}$ denote the empirical transition kernel matrix at time step $h$, associated with the joint policy $\pi_{-i}$ and agent $i$. Similarly, $\widehat{P}_{h,s,a_{i}}^{\pi_{-i}}\in\mathbb{R}^{1\times S}$ represents the row corresponding to the state-action pair $(s,a_{i})$.
- •
Define $\underline{P}_{h}^{\pi}\in\mathbb{R}^{S\times S}$ as $\underline{P}_{h}^{\pi}:=\Pi_{h}^{\pi}P_{h}^{0}$, where $\Pi_{h}^{\pi}$ is the policy matrix at time step $h$ under joint policy $\pi$.
- •
Define $\underline{\widehat{P}}_{i,h}^{\pi}\in\mathbb{R}^{S\times S}$ as $\underline{\widehat{P}}_{i,h}^{\pi}:=\Pi_{h}^{\pi_{i}}\widehat{P}_{i,h}^{\pi_{-i}}$, where $\Pi_{h}^{\pi_{i}}$ denotes the policy matrix at time step $h$ under policy $\pi_{i}$.

We introduce matrix notations for transitions that are associated not only with the nominal transition and policy $\pi$, but also with value functions:

- •
For time step $h\in[H]$, joint policy $\pi$, and a value vector $V\in\mathbb{R}^{S}$, we define $P_{i,h}^{\pi_{-i},V}\in\mathbb{R}^{SA_{i}\times S}$ as the matrix representing the worst-case transition probability kernel within the uncertainty set for agent $i$, centered around the nominal kernel. The row corresponding to the state-action pair $(s,a_{i})$ in $P_{i,h}^{\pi_{-i},V}$, denoted as $P_{i,h,s,a_{i}}^{\pi_{-i},V}\in\mathbb{R}^{S}$, is given by:
$\displaystyle P_{i,h,s,a_{-i}}^{\pi_{-i},V}$
$\displaystyle=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}(P_{h,s,a_{i}}^{\pi_{-i}})}\mathcal{P}V.$
(32a)
We also define the transition matrices for specific value vectors as:
$\displaystyle P_{i,h}^{\pi,V}:=P_{i,h}^{\pi_{-i},V_{i,h+1}^{\pi,\sigma_{i}}}\quad\text{and}\quad P_{i,h,s,a_{i}}^{\pi,V}:=P_{i,h,s,a_{i}}^{\pi_{-i},V_{i,h+1}^{\pi,\sigma_{i}}}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}(P_{h,s,a_{i}}^{\pi_{-i}})}\mathcal{P}V_{i,h+1}^{\pi,\sigma_{i}}.$
(32b)
Finally, we define square matrices $\underline{P}_{i,h}^{\pi,V}\in\mathbb{R}^{S\times S}$ as:
$\underline{P}_{i,h}^{\pi,V}:=\Pi_{h}^{\pi_{i}}P_{i,h}^{\pi_{-i},V}.$
- •
By replacing the nominal transition kernel with the empirical transition kernel, we similarly define $\widehat{P}_{i,h}^{\pi_{-i},V}$ as the worst-case probability transition kernel within the uncertainty set for agent $i$, centered around the empirical kernel $\widehat{P}_{i,h}^{\pi_{-i}}$. The row corresponding to the state-action pair $(s,a_{i})$ in $\widehat{P}_{i,h}^{\pi_{-i},V}$ is denoted as $\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i},V}\in\mathbb{R}^{S}$ and is defined as:
$\displaystyle\widehat{P}_{i,h,s,a_{-i}}^{\pi_{-i},V}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}(\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i}})}\mathcal{P}V.$
(32c)
The transition matrices $\widehat{P}_{i,h}^{\pi,V}$ for specific value vectors are defined as:
$\displaystyle\widehat{P}_{i,h}^{\pi,V}:=\widehat{P}_{i,h}^{\pi_{-i},V_{i,h+1}^{\pi,\sigma_{i}}}\quad\text{and}\quad\widehat{P}_{i,h,s,a_{i}}^{\pi,V}:=\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i},V_{i,h+1}^{\pi,\sigma_{i}}}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}(\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i}})}\mathcal{P}V_{i,h+1}^{\pi,\sigma_{i}},$
(32d)
Additionally, we define square matrices $\underline{\widehat{P}}_{i,h}^{\pi,V}\in\mathbb{R}^{S\times S}$ as:
$\underline{\widehat{P}}_{i,h}^{\pi,V}:=\Pi_{h}^{\pi_{i}}\widehat{P}_{i,h}^{\pi_{-i},V}.$

##### Variance.

We now introduce notations for variance corresponding to a specific probability distribution. For a probability vector $P\in\mathbb{R}^{1\times S}$ and a vector $V\in\mathbb{R}^{S}$, we denote the variance of $V$ with respect to $P$ as $\mathsf{Var}_{P}(V)$, defined as:

$$ $\displaystyle\mathsf{Var}_{P}(V):=P(V\circ V)-(PV)\circ(PV),$ (33) $$

Additionally, for a transition kernel $P^{\pi_{-i}}\in\mathbb{R}^{SA_{i}\times S}$ and a vector $V\in\mathbb{R}^{S}$, we define $\mathsf{Var}_{P^{\pi_{-i}}}(V)\in\mathbb{R}^{SA_{i}}$ as a vector of variances. The entry corresponding to $(s,a_{i})$ in $\mathsf{Var}_{P^{\pi_{-i}}}(V)$ is given by:

$$ $\displaystyle\mathsf{Var}_{P^{\pi_{-i}}}(s,a_{i}):=\mathsf{Var}_{P_{s,a_{i}}^{\pi_{-i}}}(V),$ (34) $$

where $P_{s,a_{i}}^{\pi_{-i}}$ denotes the row of the transition matrix corresponding to state $s$ and action $a_{i}$.

### A.2 Preliminary facts about FTRL

Our proposed algorithm (see Algorithm [2](#alg2)) is inspired by online adversarial learning. In this section, we introduce the formulation of online learning and review key aspects of a widely-used algorithm, the Follow-the-Regularized-Leader (FTRL) algorithm.

##### Problem setting: online learning for weighted average loss.

We consider an online learning problem over $K$ steps, commonly found in adversarial learning settings (Lattimore and Szepesvári,, ). The learner is presented with an action set $\mathcal{A}$, and loss functions $f_{1},\ldots,f_{K}:\mathcal{A}\to\mathbb{R}_{\geq 0}$ are provided for each step. At each time step $k$, the learner selects a distribution over the action set, $\pi_{k}\in\Delta(\mathcal{A})$, and observes the loss function $f_{k}(\pi_{k})$. The goal of the learner is to minimize the weighted average loss over the $K$ steps, which is defined as:
$L^{K}=\sum_{k=1}^{K}\alpha_{k}^{K}f_{k}(\pi_{k}).$
To evaluate the learner’s performance, the regret for the online learning process is defined as:

$$ $\displaystyle R^{K}=\sum_{k=1}^{K}\alpha_{k}^{K}f_{k}(\pi_{k})-\left[\min_{\pi\in\Delta(\mathcal{A})}\sum_{k=1}^{K}\alpha_{k}^{K}f_{k}(\pi)\right].$ (35) $$

##### FTRL and its regret bound.

A widely-used method for solving the online learning problem described above is the Follow-the-Regularized-Leader (FTRL) algorithm, introduced by Shalev-Shwartz and Singer, (); Shalev-Shwartz, (). At each step $k+1$, the learner selects a soft-greedy action by solving:

$$ $\displaystyle\pi_{k+1}=\arg\min_{\pi\in\Delta(\mathcal{A})}\left[\sum_{i=1}^{k}\alpha_{i}^{k}f_{i}(\pi)+F_{k}(\pi)\right],\quad k=1,2,\ldots,$ (36) $$

where $F_{k}(\pi)$ represents a convex regularization function. The following theorem provides a refined regret bound for the FTRL algorithm when the loss function is linear with respect to the policy.

###### Theorem 3 (Theorem 3 in Li et al., ( 2022 ) ) .

For all $k\in[K]$ and policy $\pi$, the loss function is defined as $f_{k}(\pi)=\left<\pi_{k},l_{k}\right>$, where $l_{k}\in\mathbb{R}^{|\mathcal{A}|}$ represents a loss vector. The learner’s choice $\pi_{k+1}$ in episode $k+1$ is updated according to the FTRL algorithm:

$$ $\displaystyle\pi_{k+1}(a)=\arg\min_{\pi\in\Delta(\mathcal{A})}\left\{\left<\pi,L_{k}\right>+F_{k}(\pi)\right\}=\frac{\exp\big{(}-\eta_{k+1}L_{k}(a)\big{)}}{\sum_{a^{\prime}\in\mathcal{A}}\exp\big{(}-\eta_{k+1}L_{k}(a^{\prime})\big{)}},\quad\text{for all }a\in\mathcal{A},$ (37) $$

where the regularization function is given by $F_{k}(\pi)=\sum_{a\in\mathcal{A}}\frac{1}{\eta_{k+1}}\pi(a)\log(\pi(a))$. Suppose $0<\alpha_{1}\leq 1$ and $\eta_{1}=\eta_{2}(1-\alpha_{1})$, and for all $k\geq 2$, assume $0<\alpha_{k}<1$ and $0<\eta_{k+1}(1-\alpha_{k})\leq\eta_{k}$. Define:

$$ $\displaystyle\widehat{\eta}_{k}:=\begin{cases}\eta_{2},&\text{if }k=1,\\ \frac{\eta_{k}}{1-\alpha_{k}},&\text{if }k>1.\end{cases}$ (38) $$

Then, the regret of the FTRL algorithm is bounded by:

$$ $\displaystyle R_{n}$ $\displaystyle\leq\max_{a\in\mathcal{A}}\left[\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{k},l_{k}\right>-\sum_{k=1}^{K}\alpha_{k}^{K}l_{k}(a)\right]$ $\displaystyle\leq\frac{5}{3}\sum_{k=1}^{n}\alpha_{k}^{n}\widehat{\eta}_{k}\alpha_{k}\mathsf{Var}_{\pi_{k}}(l_{k})+\frac{\log A}{\eta_{n+1}}+3\sum_{k=1}^{n}\alpha_{k}^{n}\widehat{\eta}_{k}^{2}\alpha_{k}^{2}\|l_{k}\|_{\infty}^{3}\mathbb{I}\left(\widehat{\eta}_{k}\alpha_{k}\|l_{k}\|_{\infty}>\frac{1}{3}\right).$ (39) $$

###### Theorem 3 (Theorem 3 in Li et al., ( 2022 ) ) .

## Appendix B Proof for Section 3

### B.1 Proof of Theorem 1

##### Step 1: preliminaries.

First, we introduce some useful definition and existing facts that are standard in real analysis and game theory literature.

###### Definition 2 (Upper semi-continuous) .

A point-to-set mapping $x\in\mathcal{X}\mapsto\phi(x)\in\mathcal{Y}$ is upper semi-continuous if $\lim_{n\rightarrow\infty}x^{n}=x_{0},y^{n}\in\phi(x^{n}),\lim_{n\rightarrow\infty}y^{n}=y_{0}$ imply that $y^{0}\in\phi(x_{0})$.

###### Theorem 4 (Kakutani’s fixed point Theorem (Kakutani,, 1941 ) ) .

If $X$ is a closed, bounded, and convex set in a Euclidean space, and $\phi$ is a upper semi-continuous correspondence mapping $X$ into the family of all closed convex subsets of $X$, then there exists $x\in X$ so that $x\in\phi(x)$.

###### Definition 2 (Upper semi-continuous) .

###### Theorem 4 (Kakutani’s fixed point Theorem (Kakutani,, 1941 ) ) .

##### Step 2: constructing an auxiliary single-step game.

Focusing on finite-horizon RMG $\mathcal{MG}_{\mathsf{rob}}=\big{\{}{\mathcal{S}},\{\mathcal{A}_{i}\}_{1\leq i\leq n},\{\mathcal{U}_{\rho}^{\sigma_{i}}(P^{0})\}_{1\leq i\leq n},r,H\big{\}}$, we shall verify the theorem by firstly consider a one-step game and then apply the results recursively to the sequential Markov games.

Without loss of generality, we focus on any of the steps $h\in[H]$ and construct an auxiliary one-step game. Towards this, we first introduce a fixed value function $V_{i,h+1}\in\mathbb{R}^{S}$ with $0\leq V_{i,h+1}\leq H$ for the $i$-th agent, representing the possible value function obtained at the next time step $h+1$. Focusing on time step $h$, for any joint product policy $\pi:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$, we abuse the notation defined in ([14](#S3.E14)) to denote the expected nominal transition kernel over each $(s,a_{i})$ as:

$$ $\displaystyle P^{\pi_{-i}}_{h,s,a_{i}}$ $\displaystyle=\mathbb{E}_{\pi(\bm{a}_{-i}\,|\,s,a_{i})}\left[P^{0}_{h,s,(a_{i},\bm{a}_{-i})}\right]=\mathbb{E}_{\pi_{-i}(\bm{a}_{-i}\,|\,s)}\left[P^{0}_{h,s,(a_{i},\bm{a}_{-i})}\right].$ (40) $$

Armed with this, for any joint product policy $\pi:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$, we can define the payoffs to maximize for the players as below:

$$ $\displaystyle\forall s\in{\mathcal{S}}:\quad f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V_{i,h+1})=\mathbb{E}_{\bm{a}\sim\pi(s)}[r_{i,h}(s,\bm{a})]+\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\left[\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\right],$ (41) $$

which is defined analogous to the robust Bellman equation (cf. ([13](#S3.E13))) by replacing a real robust value function vector (associated with some policy) to some fixed vector $V_{i,h+1}$.

Now we are ready to introduce the following useful mapping: for any $\pi:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$,

$$ $\displaystyle\phi(\pi)\coloneqq\left\{u\,|\,u_{i}(s)\in\mathrm{argmax}_{\pi_{i}^{\prime}(s)\in\Delta(\mathcal{A}_{i})}\;f_{i,s}(\pi_{i}^{\prime}(s),\pi_{-i}(s);V_{i,h+1}),\forall(i,s)\in[n]\times{\mathcal{S}}\right\}.$ (42) $$

##### Step 3: the existence of NE of the auxiliary game.

To apply Theorem [4](#Thmtheorem4), there are three required conditions. First, we know that the space of product policy is $X=\{\pi:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(A_{i})\}$ is a closed, bounded and convex set in Euclidean space.

- •
Verifying that $\phi(\pi)$ is an upper semi-continuous correspondence.
Before starting, we introduce the following two useful lemmas with the proof postponed to Appendix [B.2.2](#A2.SS2.SSS2) and [B.2.3](#A2.SS2.SSS3).
Lemma 2.
The set of function $\left\{f_{i,s}(\pi_{i}^{\prime}(s),\pi_{-i}(s);V_{i,h+1}),0\leq V_{i,h+1})\leq H\right\}$ is equicontinuous with respect to $\pi_{i}^{\prime}(s),\pi_{-i}(s)$ for all $(i,s)\in[n]\times{\mathcal{S}}$.
Lemma 3.
For any $i\in[n]$ and then $x_{-i}:{\mathcal{S}}\mapsto\prod_{j\neq i,j\in[n]}\Delta(\mathcal{A}_{j})$, the functions
$\displaystyle\forall s\in{\mathcal{S}}:\quad g_{i,s}(x_{-i}(s),V_{i,h+1})\coloneqq\mathrm{max}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),x_{-i}(s);V_{i,h+1})$
(43)
are continuous with respect to $x_{-i}(s)$ and the set $\{g_{i,s}(\cdot,V)|V\in\mathbb{R}^{S},0\leq V\leq H\}$ is equicontinuous.
Armed with above lemmas, we are in the position to prove this condition.
We suppose there are two sequence $\lim_{n\rightarrow\infty}x^{n}=x^{0},y^{n}\in\phi(x^{n}),\lim_{n\rightarrow\infty}y^{n}=y^{0}$.
Recall the definition of a upper semi-continuous correspondence (cf. Definition [2](#Thmdefinition2)), we are supposed to show that $y^{0}\in\phi(x^{0})$, i.e.,
$\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad f_{i,s}(y^{0}_{i}(s),x^{0}_{-i}(s);V_{i,h+1})=\mathrm{max}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),x^{0}_{-i}(s);V_{i,h+1}).$
(44)
Towards this, we have
$\displaystyle|f_{i,s}(y^{0}_{i}(s),x^{0}_{-i}(s);V_{i,h+1})-g_{i,s}(x^{0}_{-i}(s),V_{i,h+1})|$
$\displaystyle\leq|f_{i,s}(y^{0}_{i}(s),x^{0}_{-i}(s);V_{i,h+1})-f_{i,s}(y^{n}_{i}(s),x^{n}_{-i}(s);V_{i,h+1})|$
$\displaystyle\quad+|f_{i,s}(y^{n}_{i}(s),x^{n}_{-i}(s);V_{i,h+1})-g_{i,s}(x^{0}_{-i}(s),V_{i,h+1})|$
$\displaystyle\overset{\mathrm{(i)}}{=}|f_{i,s}(y^{0}_{i}(s),x^{0}_{-i}(s);V_{i,h+1})-f_{i,s}(y^{n}_{i}(s),x^{n}_{-i}(s);V_{i,h+1})|+|g_{i,s}(x^{n}_{-i}(s),V_{i,h+1})-g_{i,s}(x^{0}_{-i}(s),V_{i,h+1})|$
$\displaystyle\qquad\rightarrow 0\qquad\text{as }\quad n\rightarrow\infty,$
(45)
where the first inequality follows from the triangle inequality, (i) holds by the assumption $y^{n}\in\phi(x^{n})$ so that $f_{i,s}(y^{n}_{i}(s),x^{n}_{-i}(s);V_{i,h+1})=\mathrm{max}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),x_{-i}^{n}(s);V_{i,h+1})$, and the last line can be verified by the continuity implied by Lemma [2](#Thmlemma2) and Lemma [3](#Thmlemma3).
- •
Verifying $\phi(\pi)$ is convex for any $\pi\in X$.
Finally, we gonna work on the convexity of $\phi(\pi)$ for any $\pi\in X$. To begin with, by the definition of $\phi(\pi)$ in ([42](#A2.E42)), we know that $\phi(\pi)\subseteq X$ and the maximum of the continuous function $f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V_{i,h+1})$ (cf. Lemma [2](#Thmlemma2)) on a compact set exists, i.e., $\phi(x)\neq\emptyset$.
Suppose there exists two Nash equilibrium $z:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i}),v:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$ and $z,v\in\phi(\pi)$. Then we have that for any $(i,s)\in[n]\times{\mathcal{S}}$,
$\displaystyle f_{i,s}(z_{i}(s),\pi_{-i}(s);V_{i,h+1})=f_{i,s}(v_{i}(s),\pi_{-i}(s);V_{i,h+1})=\max_{u_{i}(s)\in\Delta(\mathcal{A}_{i})}f_{i,s}(u_{i}(s),\pi_{-i}(s);V_{i,h+1}).$
(46)
To continue, for any $0\leq\lambda\leq 1$, one has
$\displaystyle\max_{u_{i}(s)\in\Delta(\mathcal{A}_{i})}f_{i,s}(u_{i}(s),\pi_{-i}(s);V_{i,h+1})=\lambda f_{i,s}(z_{i}(s),\pi_{-i}(s);V_{i,h+1})+(1-\lambda)f_{i,s}(v_{i}(s),\pi_{-i}(s);V_{i,h+1})$
$\displaystyle=\lambda\bigg{(}\mathbb{E}_{a_{i}\sim z_{i}(s)}\left[r^{\pi_{-i}}_{i,h}(s,a_{i})\right]+\mathbb{E}_{a_{i}\sim z_{i}(s)}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}\bigg{)}$
$\displaystyle\quad+(1-\lambda)\bigg{(}\mathbb{E}_{a_{i}\sim v_{i}(s)}\left[r^{\pi_{-i}}_{i,h}(s,a_{i})\right]+\mathbb{E}_{a_{i}\sim v_{i}(s)}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}\bigg{)}$
(47)
$\displaystyle=\mathbb{E}_{a_{i}\sim[\lambda z_{i}(s)+(1-\lambda)v_{i}(s)]}\left[r^{\pi_{-i}}_{i,h}(s,a_{i})\right]+\mathbb{E}_{a_{i}\sim[\lambda z_{i}(s)+(1-\lambda)v_{i}(s)]}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}$
$\displaystyle=f_{i,s}(\lambda z_{i}(s)+(1-\lambda)v_{i}(s),\pi_{-i}(s);V_{i,h+1}).$
(48)
where we denote $r_{i,h}^{\pi_{-i}}(s,a_{i})\coloneqq\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i}(s)}\left[r_{i,h}(s,(a_{i},\bm{a}_{-i}))\right]$.
Hence, we show that $\lambda z_{i}(s)+(1-\lambda)v_{i}(s)\in\phi(\pi)$ for all $(i,s)\in[n]\times{\mathcal{S}}$ and $0\leq\lambda\leq 1$, thus verify that $\phi(\pi)$ is convex for any $\pi\in X$.

###### Lemma 2 .

###### Lemma 3 .

##### Step 4: the existence of robust NE in RMGs.

Armed with above results, now we consider a general form to show that there exists a policy $\pi:[H]\times{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$ that satisfies

$$ $\displaystyle\forall(i,h,s)\in[n]\times[H]\times{\mathcal{S}}:\quad V_{i,h}^{\pi,\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s).$ (49) $$

We shall prove this by induction.

- •
The base case. Starting with the final step $h=H$, we recall that by definition,
$\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad V^{\pi,\sigma_{i}}_{i,H+1}(s)=0.$
(50)
To apply the results in the one-step game constructed in Step 2,
we consider the one-step game at $h=H$ and using the payoff function (cf. ([41](#A2.E41)))
$\displaystyle\forall s\in{\mathcal{S}}:\quad f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V^{\pi,\sigma_{i}}_{i,H+1})=\mathbb{E}_{\bm{a}\sim\pi(s)}[r_{i,h}(s,\bm{a})].$
(51)
We know that there exists a policy $\pi$ so that
$\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad V_{i,H}^{\pi,\sigma_{i}}(s)=V_{i,H}^{\star,\pi_{-i},\sigma_{i}}(s)$
(52)
by setting $\pi_{H}$ as the NE of the one-step game.
- •
Induction.
Assuming that there exists a policy $\pi$ so that for subsequent steps $h+1,\cdots,H$,
$\displaystyle\forall(i,h,s)\in[n]\times\{h+1,\cdots,H\}\times{\mathcal{S}}:\quad V_{i,h}^{\pi,\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s),$
(53)
which are achieved by determining certain policies for $\{\pi_{h+1},\pi_{h+2},\cdots,\pi_{H}\}$.
We are supposed to prove that at time step $h$, we can ensure our policy $\pi$ satisfying
$\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad V_{i,h}^{\pi,\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s)$
(54)
by choosing a proper policy $\pi_{h}$ at the time step $h$.
Towards this, it is observed that
$\displaystyle V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s)$
$\displaystyle=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}V_{i,h}^{\pi^{\prime}_{i}\times\pi_{-i},\sigma_{i}}(s)$
$\displaystyle=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}\mathbb{E}_{\bm{a}\sim\pi^{\prime}_{i,h}(s)\times\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]+\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i,h}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\pi_{i}^{\prime}\times\pi_{-i},\sigma_{i}}_{i,h+1}\bigg{]}$
$\displaystyle=\max_{\pi^{\prime}_{i,h}(s)\in\Delta(\mathcal{A}_{i})}\mathbb{E}_{\bm{a}\sim\pi^{\prime}_{i,h}(s)\times\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]$
$\displaystyle\quad+\max_{\pi^{\prime}_{i,h}(s)\in\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i,h}(s)}\max_{\pi^{\prime}_{i,h^{+}}:{\mathcal{S}}\times h^{+}\mapsto\Delta(\mathcal{A}_{i})}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\pi_{i}^{\prime}\times\pi_{-i},\sigma_{i}}_{i,h+1}\bigg{]}$
$\displaystyle=\max_{\pi^{\prime}_{i,h}(s)\in\Delta(\mathcal{A}_{i})}\left[\mathbb{E}_{\bm{a}\sim\pi^{\prime}_{i,h}(s)\times\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]+\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i,h}(s)}\left[\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\star,\pi_{-i},\sigma_{i}}_{i,h+1}\right]\right].$
(55)
where we denote $h^{+}=\{h+1,h+2,\cdots,H\}$ as the set that includes all the time steps after $h$ until the end of the episode, and the last equality follows from the fact
$\displaystyle\max_{\pi^{\prime}_{i,h^{+}}:{\mathcal{S}}\times h^{+}\mapsto\Delta(\mathcal{A}_{i})}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\pi_{i}^{\prime}\times\pi_{-i},\sigma_{i}}_{i,h+1}\bigg{]}$
$\displaystyle=\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}P\max_{\pi^{\prime}_{i,h^{+}}:{\mathcal{S}}\times h^{+}\mapsto\Delta(\mathcal{A}_{i})}V^{\pi_{i}^{\prime}\times\pi_{-i},\sigma_{i}}_{i,h+1}\bigg{]}$
$\displaystyle=\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\star,\pi_{-i},\sigma_{i}}_{i,h+1},$
(56)
which holds by the definition of $V^{\star,\pi_{-i},\sigma_{i}}_{i,h+1}$.
Now invoking the results in the auxiliary one-step game with $V_{i,h+1}=V^{\star,\pi_{-i},\sigma_{i}}_{i,h+1}$, one has that there exists a policy with $\pi_{h}$ that satisfies
$\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad V_{i,h}^{\pi,\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s).$
(57)

Combining the results in the base case and induction, we complete the proof by recursively choosing $\pi_{h}:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i})$ for $h=H,H-1,\cdots,1$ as the NE of the corresponding one-step game at time step $h$ and arrive at

$$ $\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad V_{i,1}^{\pi,\sigma_{i}}(s)=V_{i,1}^{\star,\pi_{-i},\sigma_{i}}(s).$ (58) $$

### B.2 Proof of auxiliary facts

#### B.2.1 Proof of Lemma 1

The proof is obtained by recursively showing that for each $(h,s)$, there exist a policy. Then the product policy of them will be that final policy

Without loss of generality, we consider any $i\in[n]$ with the other agents’ policy $\pi_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})$ fixed.
We shall prove this lemma by induction.

- •
The base case. Consider the base case $h=H$. Conditioned on the other agents’ policy $\pi_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{-i})$, the maximum of the robust value function of the $i$-th agent can be expressed by
$\displaystyle\forall s\in{\mathcal{S}}:\quad V_{i,H}^{\star,\pi_{-i},\sigma_{i}}(s)$
$\displaystyle=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}V_{i,H}^{\pi^{\prime}_{i}\times\pi_{-i},\sigma_{i}}(s)$
$\displaystyle=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,H}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,H}(s)}[r_{i,H}(s,\bm{a})]\right]$
$\displaystyle=\max_{\pi^{\prime}_{i,H}(s)\sim\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,H}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,H}(s)}[r_{i,H}(s,\bm{a})]\right].$
(59)
Since the maximum of the continuous function $\mathbb{E}_{a_{i}\sim\pi_{i,H}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,H}(s)}[r_{i,H}(s,\bm{a})]\right]$ on a compact set $\Delta(\mathcal{A}_{i})$ exists, by setting
$\displaystyle\forall s\in{\mathcal{S}}:\quad\widetilde{\pi}_{i,H}(s)=\mathrm{argmax}_{\pi^{\prime}_{i,H}(s)\sim\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,H}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,H}(s)}[r_{i,H}(s,\bm{a})]\right],$
(60)
we arrive at
$\displaystyle\forall s\in{\mathcal{S}}:\quad V_{i,H}^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}(s)=V_{i,H}^{\star,\pi_{-i},\sigma_{i}}(s).$
(61)
This complete the proof for the base case.
- •
Induction.
Assuming that for $t=h+1,h+2,\cdots,H$, we have
$\displaystyle\forall s\in{\mathcal{S}}:\quad V_{i,t}^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}(s)=V_{i,t}^{\star,\pi_{-i},\sigma_{i}}(s).$
(62)
Then, we want to prove for the step $h$, where the maximum of the robust value function of the $i$-th agent can be expressed as: for all $s\in{\mathcal{S}}$,
$\displaystyle V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s)$
$\displaystyle=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}V_{i,h}^{\pi^{\prime}_{i}\times\pi_{-i},\sigma_{i}}(s)$
$\displaystyle=\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}(s)}\left[\inf_{\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\star,\pi_{-i},\sigma_{i}}_{i,h+1}\right]$
$\displaystyle\overset{\mathrm{(i)}}{=}\max_{\pi^{\prime}_{i}:{\mathcal{S}}\times[H]\mapsto\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}(s)}\left[\inf_{\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}_{i,h+1}\right]$
$\displaystyle=\max_{\pi^{\prime}_{i,h}(s)\sim\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}(s)}\left[\inf_{\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}_{i,h+1}\right].$
(63)
where (i) holds by the induction assumption in ([62](#A2.E62)). Similarly to the base case, the maximum of the continuous function $\mathbb{E}_{a_{i}\sim\pi_{i,h}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}(s)}\left[\inf_{\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}_{i,h+1}\right]$ on a compact set $\Delta(\mathcal{A}_{i})$ exists. So without conflict, for all $s\in{\mathcal{S}}$, we can set
$\displaystyle\widetilde{\pi}_{i,h}(s)$
$\displaystyle=\mathrm{argmax}_{\pi^{\prime}_{i,h}(s)\sim\Delta(\mathcal{A}_{i})}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{\prime}(s)}\left[\mathbb{E}_{\bm{a}_{-i}\sim\pi_{-i,h}(s)}[r_{i,h}(s,\bm{a})]\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}(s)}\left[\inf_{\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}_{i,h+1}\right],$
(64)
since the function $\inf_{\mathcal{U}^{\sigma_{i}}_{\rho}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}_{i,h+1}$ and especially $V^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}_{i,h+1}$ are independent from the policy in the first $h$ steps ($\{\widetilde{\pi}_{i,t}(s)\}_{s\in{\mathcal{S}},t\in[h]}$).
Consequently, ([64](#A2.E64)) directly implies that
$\displaystyle\forall s\in{\mathcal{S}}:\quad V_{i,h}^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s).$
(65)

Combining the results in base case and the induction, we complete the proof by showing that

$$ $\displaystyle\forall(h,s)\in[H]\times{\mathcal{S}}:\quad V_{i,h}^{\widetilde{\pi}_{i}\times\pi_{-i},\sigma_{i}}(s)=V_{i,h}^{\star,\pi_{-i},\sigma_{i}}(s).$ (66) $$

#### B.2.2 Proof of Lemma 2

First, we define the distance between any two policy $\pi,\pi^{\prime}\in X=\{\pi:{\mathcal{S}}\mapsto\prod_{i\in[n]}\Delta(A_{i})\}$ as below:

$$ $\displaystyle d(\pi,\pi^{\prime})\coloneqq\max_{i\in[n]}\max_{(s,a_{i})\in{\mathcal{S}}\times\mathcal{A}_{i}}|\pi_{i}(a_{i}\,|\,s)-\pi^{\prime}_{i}(a_{i}\,|\,s)|.$ (67) $$

To prove the continuity, given any $\epsilon>0$, we want to show that there exists $\delta(\epsilon)>0$ such that if

$$ $\displaystyle d(\pi,\pi^{\prime})<\delta(\epsilon),$ (68) $$

then

$$ $\displaystyle\left|f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V_{i,h+1})-f_{i,s}(\pi^{\prime}_{i}(s),\pi^{\prime}_{-i}(s);V_{i,h+1})\right|<\epsilon$ (69) $$

for any fixed $\{V_{i,h+1}\}_{i\in[n]}$ with $0\leq V_{i,h+1}\leq H$ for all $i\in[n]$.
Towards this, we observe that

$$ $\displaystyle\left|f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V_{i,h+1})-f_{i,s}(\pi^{\prime}_{i}(s),\pi^{\prime}_{-i}(s);V_{i,h+1})\right|$ $\displaystyle=\bigg{|}\mathbb{E}_{\bm{a}\sim\pi(s)}[r_{i,h}(s,\bm{a})]+\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}$ $\displaystyle\quad-\mathbb{E}_{\bm{a}\sim\pi^{\prime}(s)}[r_{i,h}(s,\bm{a})]+\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i}(s)}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\big{(}P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\big{)}}PV_{i,h+1}\bigg{]}\bigg{|}$ $\displaystyle\leq\left|\mathbb{E}_{\bm{a}\sim\pi(s)}[r_{i,h}(s,\bm{a})]-\mathbb{E}_{\bm{a}\sim\pi^{\prime}(s)}[r_{i,h}(s,\bm{a})]\right|$ $\displaystyle\quad+\bigg{|}\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}-\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i}(s)}\bigg{[}\inf_{\mathcal{U}^{\sigma_{i}}\big{(}P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\big{)}}PV_{i,h+1}\bigg{]}\bigg{|}.$ (70) $$

The first term can be bounded by

$$ $\displaystyle\left|\mathbb{E}_{\bm{a}\sim\pi(s)}[r_{i,h}(s,\bm{a})]-\mathbb{E}_{\bm{a}\sim\pi^{\prime}(s)}[r_{i,h}(s,\bm{a})]\right|$ $\displaystyle\leq\sum_{\bm{a}\in\mathcal{A}}\bigg{|}\prod_{i\in[n]}\pi_{i}(a_{i}\,|\,s)-\prod_{i\in[n]}\pi^{\prime}_{i}(a_{i}\,|\,s)\bigg{|}\max_{(s,\bm{a})\in{\mathcal{S}}\times\mathcal{A}}r_{i,h}(s,\bm{a})$ $\displaystyle\leq\sum_{\bm{a}\in\mathcal{A}}\bigg{|}\prod_{i\in[n]}\pi_{i}(a_{i}\,|\,s)-\prod_{i\in[n]}\pi^{\prime}_{i}(a_{i}\,|\,s)\bigg{|},$ (71) $$

where the last inequality holds by the definition of reward function $\max_{(s,\bm{a})\in{\mathcal{S}}\times\mathcal{A}}r_{i,h}(s,\bm{a})\leq 1$ for all $(i,h)\in[n]\times[H]$. To continue, we first define the difference between $\delta_{i}(s,a_{i})\coloneqq\pi^{\prime}_{i}(a_{i}\,|\,s)-\pi_{i}(a_{i}\,|\,s)$. Therefore, we have

$$ $\displaystyle\bigg{|}\prod_{i\in[n]}\pi_{i}(a_{i}\,|\,s)-\prod_{i\in[n]}\pi^{\prime}_{i}(a_{i}\,|\,s)\bigg{|}$ $\displaystyle=\bigg{|}\prod_{i\in[n]}\pi_{i}(a_{i}\,|\,s)-\prod_{i\in[n]}(\pi_{i}(a_{i}\,|\,s)+\delta_{i}(s,a_{i}))\bigg{|}$ $\displaystyle=\bigg{|}\sum_{|\mathcal{Y}|\geq 1,\mathcal{Y}\subseteq[n]}\left(\prod_{i\in\mathcal{Y}}\delta_{i}(s,a_{i})\right)\cdot\left(\prod_{i\in\mathcal{Y}^{c}}\pi_{i}(a_{i}\,|\,s)\right)\bigg{|}$ $\displaystyle\leq\sum_{|\mathcal{Y}|\geq 1,\mathcal{Y}\subseteq[n]}\bigg{|}\left(\prod_{i\in\mathcal{Y}}\delta_{i}(s,a_{i})\right)\cdot\left(\prod_{i\in\mathcal{Y}^{c}}\pi_{i}(a_{i}\,|\,s)\right)\bigg{|}\leq(2^{n}-1)\delta(\epsilon),$ (72) $$

where the last inequality holds by ([68](#A2.E68)). Plugging ([72](#A2.E72)) back to ([71](#A2.E71)) indicates that

$$ $\displaystyle\left|\mathbb{E}_{\bm{a}\sim\pi(s)}[r_{i,h}(s,\bm{a})]-\mathbb{E}_{\bm{a}\sim\pi^{\prime}(s)}[r_{i,h}(s,\bm{a})]\right|\leq\prod_{i\in[n]}A_{i}(2^{n}-1)\delta(\epsilon).$ (73) $$

For the second term in ([70](#A2.E70)), we observe that

$$ $\displaystyle\bigg{|}\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}-\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\big{(}P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\big{)}}PV_{i,h+1}\bigg{]}\bigg{|}$ $\displaystyle\leq\bigg{|}\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}-\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}\bigg{|}$ $\displaystyle\quad+\bigg{|}\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\left(P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\right)}PV_{i,h+1}\bigg{]}-\mathbb{E}_{a_{i}\sim\pi^{\prime}_{i}(s)}\bigg{[}\inf_{P\in\mathcal{U}^{\sigma_{i}}\big{(}P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\big{)}}PV_{i,h+1}\bigg{]}\bigg{|}$ $\displaystyle\overset{\mathrm{(i)}}{\leq}\mathbb{E}_{a_{i}\sim\pi_{i}(s)}\bigg{[}\max_{\alpha\in[\min_{s}V_{i,h+1}(s),\max_{s}V_{i,h+1}(s)]}\bigg{|}\mathbb{E}_{\pi_{-i}(\bm{a}_{-i}\,|\,s)}\left[P^{0}_{h,s,(a_{i},\bm{a}_{-i})}\right]\left[V_{i,h+1}\right]_{\alpha}$ $\displaystyle\quad-\mathbb{E}_{\pi^{\prime}_{-i}(\bm{a}_{-i}\,|\,s)}\left[P^{0}_{h,s,(a_{i},\bm{a}_{-i})}\right]\left[V_{i,h+1}\right]_{\alpha}\bigg{|}\bigg{]}+\sum_{a_{i}\in\mathcal{A}_{i}}\big{|}\pi_{i}^{\prime}(a_{i}\,|\,s)-\pi_{i}(a_{i}\,|\,s)\big{|}\inf_{P\in\mathcal{U}^{\sigma_{i}}\big{(}P^{\pi^{\prime}_{-i}}_{h,s,a_{i}}\big{)}}PV_{i,h+1}$ $\displaystyle\overset{\mathrm{(ii)}}{\leq}\sum_{\bm{a}_{-i}\in\mathcal{A}_{i}}\bigg{|}\prod_{j\neq i}\pi_{j}(a_{j}\,|\,s)-\prod_{j\neq i}\pi^{\prime}_{j}(a_{j}\,|\,s)\bigg{|}H+HA_{i}\Delta(\epsilon)$ $\displaystyle\overset{\mathrm{(iii)}}{\leq}H\prod_{j\neq i,j\in[n]}A_{j}(2^{n-1}-1)\delta(\epsilon)+HA_{i}\Delta(\epsilon)\leq 2H\prod_{i\in[n]}A_{i}(2^{n}-1)\cdot\delta(\epsilon),$ (74) $$

where the first inequality holds by the triangle inequality, and (i) follows from applying the dual form of TV distance

$$ $\displaystyle\inf_{\mathcal{P}\in U^{\sigma_{i}}(P)}\mathcal{P}V=\max_{\alpha\in[\min_{s}V(s),\max_{s}V(s)]}\Big{\{}P\left[V\right]_{\alpha}-\sigma_{i}\left(\alpha-\min_{s^{\prime}}\left[V\right]_{\alpha}(s^{\prime})\right)\Big{\}},$ (75) $$

and the maximum operator is $1$-Lipschitz, (ii) arises from the fact that $\|V_{i,h+1}\|_{\infty}\leq H$, and (iii) can be verified by following the same pipeline of ([72](#A2.E72)).
Combining ([73](#A2.E73)) and ([74](#A2.E74)), one has

$$ $\displaystyle\left|f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V_{i,h+1})-f_{i,s}(\pi^{\prime}_{i}(s),\pi^{\prime}_{-i}(s);V_{i,h+1})\right|\leq 3H\prod_{i\in[n]}A_{i}(2^{n}-1)\cdot\delta(\varepsilon).$ (76) $$

Consequently, letting $\delta_{1}(\epsilon)=\frac{\min\{\epsilon,1\}}{3H\prod_{i\in[n]}A_{i}(2^{n}-1)}$, we have when $d(\pi,\pi^{\prime})<\delta_{1}(\epsilon)$,

$$ $\left|f_{i,s}(\pi_{i}(s),\pi_{-i}(s);V_{i,h+1})-f_{i,s}(\pi^{\prime}_{i}(s),\pi^{\prime}_{-i}(s);V_{i,h+1})\right|<\epsilon.$ $$

#### B.2.3 Proof of Lemma 3

Without loss of generality, we consider any $i\in[n]$. Consider $x_{-i}:{\mathcal{S}}\mapsto\prod_{j\neq i,j\in[n]}\Delta(\mathcal{A}_{j})$ and $y_{-i}:{\mathcal{S}}\mapsto\prod_{j\neq i,j\in[n]}\Delta(\mathcal{A}_{j})$. Before continuing, for all $s\in{\mathcal{S}}$, we denote

$$ $\displaystyle u_{i,s}^{\star}$ $\displaystyle\coloneqq\mathrm{argmax}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),x_{-i}(s);V_{i,h+1}),$ $\displaystyle v_{i,s}^{\star}$ $\displaystyle\coloneqq\mathrm{argmax}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),y_{-i}(s);V_{i,h+1}).$ (77) $$

Then we have for any $s\in{\mathcal{S}}$,

$$ $\displaystyle g_{i,s}(x_{-i}(s),V_{i,h+1})-g_{i,s}(y_{-i}(s),V_{i,h+1})$ $\displaystyle=\mathrm{max}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),x_{-i}(s);V_{i,h+1})-\mathrm{max}_{\pi_{i}^{\prime}\in\Delta({\mathcal{S}})}\;f_{i,s}(\pi_{i}^{\prime}(s),y_{-i}(s);V_{i,h+1})$ $\displaystyle=f_{i,s}(u_{i,s}^{\star},x_{-i}(s);V_{i,h+1})-f_{i,s}(v_{i,s}^{\star},y_{-i}(s);V_{i,h+1})$ $\displaystyle\leq f_{i,s}(u_{i,s}^{\star},x_{-i}(s);V_{i,h+1})-f_{i,s}(u_{i,s}^{\star},y_{-i}(s);V_{i,h+1})\quad\rightarrow 0\quad\text{as}\quad y_{-i}(s)\rightarrow x_{-i}(s),$ (78) $$

where the last line holds by Lemma ([2](#Thmlemma2)) which shows that the function $f_{i,s}$ is continuous.
Similarly, one has

$$ $\displaystyle g_{i,s}(x_{-i}(s),V_{i,h+1})-g_{i,s}(y_{-i}(s),V_{i,h+1})$ $\displaystyle\geq f_{i,s}(v_{i,s}^{\star},x_{-i}(s);V_{i,h+1})-f_{i,s}(u_{i,s}^{\star},y_{-i}(s);V_{i,h+1})\quad\rightarrow 0\quad\text{as}\quad y_{-i}(s)\rightarrow x_{-i}(s).$ (79) $$

We complete the proof by showing that

$$ $\displaystyle|g_{i,s}(x_{-i}(s),V_{i,h+1})-g_{i,s}(y_{-i}(s),V_{i,h+1})|\rightarrow 0\quad\text{as}\quad y_{-i}(s)\rightarrow x_{-i}(s).$ (80) $$

## Appendix C Proof of Theorem 2

We will present the proof of Theorem [2](#Thmtheorem2) by first outlining the proof structure, followed by a step-by-step explanation of the key components. Auxiliary proofs will be provided at the end of this section.

### C.1 Proof pipeline

To proof Theorem [2](#Thmtheorem2), recall the goal is to show that

$$ $\displaystyle\forall(i,s)\in[n]\times{\mathcal{S}}:\quad\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\star,\pi_{-i},\sigma_{i}}(s)\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\pi,\sigma_{i}}(s)\right]\leq\varepsilon,$ (81) $$

where $\widehat{\xi}=\{\widehat{\xi}_{h}\}_{h\in[H]}$ is the output distribution over the set of policies $\{\pi_{h}^{k}=(\pi_{1,h}^{k}\times\cdots\times\pi_{n,h}^{k})\}_{k\in[K],h\in[H]}$ from Algorithm [2](#alg2). Namely, $\pi\sim\widehat{\xi}$ means

$$ $\displaystyle\forall h\in[H]:\quad\pi_{h}\sim\widehat{\xi}_{h},\quad\text{ where }\quad\widehat{\xi}_{h}(\pi_{h}^{k})=\alpha_{k}^{K}.$ (82) $$

We first introduce the best-response policy for player $i$:

$$ $\displaystyle\tilde{\pi}_{i}^{\star}=[\tilde{\pi}_{i,h}^{\star}]_{h\in[H]}:=\arg\max_{\pi_{i}^{\prime}:\mathcal{S}\times[H]\to\Delta(\mathcal{A}_{i})}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\pi_{i}^{\prime},\pi_{-i}}\right].$ $$

Recall that value function $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\pi,\sigma_{i}}\right]$ satisfies the following Bellman equation for all $(i,s,h)\in[n]\times\mathcal{S}\times[H]$:

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,H+1}^{\pi,\sigma_{i}}(s)\right]$ $\displaystyle=0,$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\pi,\sigma_{i}}(s)\right]$ $\displaystyle=\mathbb{E}_{\pi\sim\widehat{\xi}}\left\{\sum_{\mathbf{a}\in\mathcal{A}}\pi_{h}(\mathbf{a}\mid s)r_{i,h}(s,\mathbf{a})+\mathbb{E}_{a_{i}\sim\pi_{i,h}}\left[\inf_{\mathcal{P}\in\mathcal{U}_{i}^{\sigma_{i}}\left(P_{h,s,a_{i}}^{\pi_{-i}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h+1}^{\pi,\sigma_{i}}\right]\right]\right\},$ $\displaystyle=\sum_{k=1}^{K}\sum_{\mathbf{a}\in\mathcal{A}}\alpha_{k}^{K}\pi_{h}^{k}(\mathbf{a}\mid s)r_{i,h}(s,\mathbf{a})+\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[\inf_{\mathcal{P}\in\mathcal{U}_{i}^{\sigma_{i}}\left(P_{h,s,a_{i}}^{\pi_{-i}^{k}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h+1}^{\pi,\sigma_{i}}\right]\right],$ $$

where $P_{h,s,a_{i}}^{\pi_{-i}^{k}}$ is defined as:

$$ $\displaystyle P_{h,s,a_{i}}^{\pi_{-i}^{k}}=\mathbb{E}_{\mathbf{a}_{-i}\sim\pi_{-i,h}^{k}(\cdot\mid s)}\left[P_{h,s,(a_{i},\mathbf{a}_{-i})}^{0}\right]=\sum_{\mathbf{a}_{-i}\in\mathcal{A}_{-i}}\pi_{-i,h}^{k}(\mathbf{a}_{-i}\mid s)\left[P_{h,s,(a_{i},\mathbf{a}_{-i})}^{0}\right].$ $$

We decompose the error in the value functions as follows:

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\star,\pi_{-i}}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\pi}\right]$ (83) $\displaystyle\leq\underbrace{\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\star,\pi_{-i}}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\tilde{\pi}_{i}^{\star},\pi_{-i}}\right]}_{A}+\underbrace{\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\star,\pi_{-i}}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]}_{B}+\underbrace{\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\pi}\right]}_{C}.$ $$

We define the following auxiliary value functions for all $s\in\mathcal{S}$:

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}(s)\right]=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})\right]+\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h+1}\right]\right],$ (84a) $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\tilde{\pi}_{i}^{\star},\pi_{-i}}_{i,h}(s)\right]=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\tilde{\pi}_{i,h}^{\star}(s)}\left[r_{i,h}^{k}(s,a_{i})\right]+\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\tilde{\pi}_{i,h}^{\star}(s)}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\tilde{\pi}_{i}^{\star},\pi_{-i}}_{i,h+1}\right]\right],$ (84b) $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}(s)\right]=\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}\left[r_{i,h}^{k}(s,a_{i})+\left(\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h+1}\right]\right)\right],$ (84c) $$

where for all $s\in\mathcal{S},$ we also have

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,H+1}(s)\right]=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\tilde{\pi}_{i}^{\star},\pi_{-i}}_{i,H+1}(s)\right]=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,H+1}(s)\right]=0$ $$

Here, we use the fact that $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}\right]\geq\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\tilde{\pi}_{i}^{\star},\pi_{-i}}_{i,h}(s)\right]$.
Using the error decomposition in ([83](#A3.E83)), we will now individually bound the three terms, $A$, $B$, and $C$, in the following sections.

### C.2 Controlling B: adversarial online learning

#### C.2.1 Step 1: showing that V ^ i , h subscript ^ 𝑉 𝑖 ℎ \widehat{V}_{i,h} is an entry-wise upper bound on 𝔼 π ∼ ξ ^ ​ [ V ¯ i , h ⋆ , π − i ] subscript 𝔼 similar-to 𝜋 ^ 𝜉 delimited-[] subscript superscript ¯ 𝑉 ⋆ subscript 𝜋 𝑖 𝑖 ℎ \mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}\right]

The following lemma demonstrates that the value estimate $\widehat{V}_{i,h}$ for the $i^{\text{th}}$ player serves as an optimistic estimate of the auxiliary value $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}\right]$, as defined in ([84](#A3.E84)).

###### Lemma 4 .

With probability at least $1-\delta$, it holds that

$$ $\displaystyle\widehat{V}_{i,h}\geq\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}\right],\qquad\text{for all $(i,h)\in[n]\times[H]$.}$ $$

###### Proof.

See Appendix [C.4.1](#A3.SS4.SSS1)
∎

The following lemma demonstrates that the value estimate $\widehat{V}_{i,h}$ for the $i^{\text{th}}$ player serves as an optimistic estimate of the auxiliary value $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}\right]$, as defined in ([84](#A3.E84)).

###### Lemma 5 .

For value vector $\widehat{V}_{i,h}$ and $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]$, it holds that

$$ $\displaystyle\widehat{V}_{i,h}\geq\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right],\qquad\text{for all $(i,h)\in[n]\times[H]$.}$ $$

###### Proof.

See Appendix [C.4.2](#A3.SS4.SSS2)
∎

###### Lemma 4 .

###### Proof.

###### Lemma 5 .

###### Proof.

#### C.2.2 Step 2: constructing recursion

To begin with, according to the definition of $\widehat{V}_{i,h}(s)$ and $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$, we have

$$ $\displaystyle\widehat{V}_{i,h}(s)-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle=\min\left\{\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}(s),H-h+1\right\}$ $\displaystyle\quad-\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right]$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}(s)$ $\displaystyle\quad-\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right]$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}(s)-\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right]$ (85) $$

To simplify the notations, we define transition kernel associated estimated value function similarly as ([32](#A1.E32)). For all $k\in[K]$, we define matrix notations $\widehat{P}_{i,h}^{\pi^{k},\widehat{V}}$ and $\widehat{P}_{i,h}^{\hat{\pi}^{k},\overline{V}}$ as:

$$ $\displaystyle\widehat{P}_{i,h}^{\pi^{k},\widehat{V}}:=\widehat{P}_{i,h}^{\pi_{-i}^{k},\widehat{V}_{i,h+1}}\quad\text{and}\quad\widehat{P}_{i,h,s,a_{i}}^{\pi^{k},\widehat{V}}:=\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i}^{k},\widehat{V}_{i,h+1}}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}\left(\widehat{P}_{i,h,s,a_{i}}^{\pi^{k}_{-i}}\right)}\mathcal{P}\widehat{V}_{i,h+1},$ $\displaystyle\widehat{P}_{i,h}^{\pi^{k},\overline{V}}:=\widehat{P}_{i,h}^{\pi_{-i}^{k},\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]}\quad\text{and}\quad\widehat{P}_{i,h,s,a_{i}}^{\pi^{k},\overline{V}}:=\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i}^{k},\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}\left(\widehat{P}_{i,h,s,a_{i}}^{\pi^{k}_{-i}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right].$ $$

Additionally, we define square matrices $\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\in\mathbb{R}^{S\times S}$ and $\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}\in\mathbb{R}^{S\times S}$ as: $\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}:=\Pi_{h}^{\pi_{i}^{k}}\widehat{P}_{i,h}^{\pi_{-i}^{k},\overline{V}}$ and $\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}:=\Pi_{h}^{\pi_{i}^{k}}\widehat{P}_{i,h}^{\pi_{-i}^{k},\widehat{V}}$.
We rewrite the result of ([85](#A3.E85)) in a vector form, we can obtain that

$$ $\displaystyle\widehat{V}_{i,h}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\Pi_{h}^{\pi_{i}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i}^{k}}\right)}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}-\sum_{k=1}^{K}\alpha_{k}^{K}\Pi_{h}^{\pi_{i}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(\widehat{P}_{i,h,s,a_{i}}^{\pi_{-i}^{k}}\right)}\mathcal{P}\overline{V}_{i,h+1}^{\pi}\right]$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}\widehat{V}_{i,h+1}+\beta_{i,h}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\widehat{V}_{i,h+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)+\beta_{i,h}.$ $$

To continue, we first introduce an lemma of the upper bound for bonus vector $\beta_{i,h}$.

###### Lemma 6 .

The bonus vector $\beta_{i,h}$ is bounded by the following inequality:

$$ $\displaystyle\beta_{i,h}\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\left(H\cdot 1+\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\widehat{V}_{i,h+1}\right)$ $$

###### Proof.

See Appendix [C.4.3](#A3.SS4.SSS3)
∎

To proceed, we introduce some notations for convenience. Let $e_{s}$ denote the $S$-dimensional standard basis vector, with support on the $s$-th element. Additionally, we define:

$$ $\displaystyle b_{h}^{h}=e_{s}\quad\text{and}\quad b_{h}^{j}=e_{s}^{\top}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,r}^{\pi^{k},\overline{V}}\right)\right],\quad\forall j=h+1,\dots,H.$ (86) $$

Armed with above notations and fact, for any $s\in\mathcal{S}$, we have

$$ $\displaystyle\widehat{V}_{i,h}(s)-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}(s)\right]=\left<e_{s},\widehat{V}_{i,h}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]\right>=\sum_{j=h}^{H}\left<b_{h}^{j},\beta_{i,j}\right>$ $\displaystyle\leq\sum_{j=h}^{H}\left<b_{h}^{j},3c_{\mathsf{b}}H\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}1\right>+\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\widehat{V}_{i,j+1}\right>$ $\displaystyle=3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}+3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\widehat{V}_{i,j+1}\right>.$ (87) $$

With elementary inequality $\sqrt{\mathsf{Var}_{P}(V+V^{\prime})}\leq\sqrt{\mathsf{Var}_{P}(V)}+\sqrt{\mathsf{Var}_{P}(V^{\prime})}$ for any transition kernel $P\in\mathbb{R}^{S}$ and vector $V,V^{\prime}\in\mathbb{R}^{S}$, we further decompose ([87](#A3.E87)) as

$$ $\displaystyle\widehat{V}_{i,h}(s)-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}+3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\widehat{V}_{i,j+1}\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}+3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{k=1}^{K}\alpha_{k}^{K}\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\left(\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\quad+3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{k=1}^{K}\alpha_{k}^{K}\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}+\underbrace{3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\left(\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>}_{\mathcal{D}_{1}}$ $\displaystyle\quad+\underbrace{3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>}_{\mathcal{D}_{2}}$ $\displaystyle\quad+\underbrace{3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>}_{\mathcal{D}_{3}}.$ $$

We now control the three terms $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{3}$ separately.

###### Lemma 6 .

###### Proof.

##### Controlling 𝒟 1 subscript 𝒟 1 \mathcal{D}_{1} .

We can directly obtain the following upper bound on $\mathcal{D}_{1}$:

$$ $\displaystyle\mathcal{D}_{1}$ $\displaystyle=3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\left(\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\left\lVert\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\left(\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right\rVert_{\infty}\cdot 1\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\left\lVert\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right\rVert_{\infty}^{2}\cdot 1\right>$ $\displaystyle\overset{\mathsf{(i)}}{\leq}3c_{\mathsf{b}}\sqrt{\frac{H\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\left\lVert\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right\rVert_{\infty}\cdot 1\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\max_{h\leq j\leq H}\left\lVert\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right\rVert_{\infty}$ (88) $$

where $\mathsf{(i)}$ follows from the elementary upper bound $\left\lVert\widehat{V}_{i,j+1}\right\rVert_{\infty}\leq H$, $\left\lVert\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right\rVert_{\infty}\leq H$ for all $h\leq j\leq H$.

Before deriving the upper bounds for the terms $\mathcal{D}_{2}$ and $\mathcal{D}_{3}$, we first introduce the following auxiliary lemmas, which will be instrumental in the subsequent derivation.

###### Lemma 7 .

For all $(i,h)\in[n]\times[H]$, the estimated robust value function $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]$ satisfies the following inequality:

$$ $\displaystyle\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ $$

###### Proof.

See Appendix [C.4.4](#A3.SS4.SSS4).
∎

With Lemma [7](#Thmlemma7), we have the following lemma on variance base on different transition probability in the same uncertainty set, and we leave the proof to Appendix [C.4.5](#A3.SS4.SSS5).

###### Lemma 8 .

For a transition kernel $P^{\prime}\in\mathbb{R}^{S}$ and any $\widetilde{P}\in\mathbb{R}^{S}$ such that $\widetilde{P}\in\mathcal{U}^{\sigma_{i}}(P^{\prime})$, the following bound holds for all $(i,h,)\in[n]\times[H]$:

$$ $\displaystyle\left|\mathsf{Var}_{P^{\prime}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]\right)-\mathsf{Var}_{\widetilde{P}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]\right)\right|\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ (89a) $$

###### Lemma 7 .

###### Proof.

###### Lemma 8 .

##### Controlling 𝒟 2 subscript 𝒟 2 \mathcal{D}_{2} .

We can directly apply Lemma [8](#Thmlemma8) and arrive at

$$ $\displaystyle\left|\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)\right|$ $\displaystyle\leq\left|\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)\right|$ $\displaystyle\quad+\left|\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)\right|$ $\displaystyle\leq 2\min\left\{\frac{1}{\sigma_{i}},H\right\}.$ $$

We insert ([C.2.2](#A3.Ex1a)) back to the expression of $\mathcal{D}_{2}$, and we can obtain that

$$ $\displaystyle\mathcal{D}_{2}$ $\displaystyle=3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\widehat{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\left<b_{h}^{j},2\min\left\{\frac{1}{\sigma_{i}},H\right\}1\right>$ $\displaystyle=6c_{\mathsf{b}}\sqrt{\frac{H\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\min\left\{\frac{1}{\sigma_{i}},H\right\}.$ (90) $$

##### Controlling 𝒟 3 subscript 𝒟 3 \mathcal{D}_{3} .

We first apply Lemma [12](#Thmlemma12), and we can directly deduce that

$$ $\displaystyle\mathcal{D}_{3}$ $\displaystyle=3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\sum_{k=1}^{K}\alpha_{k}^{K}\left<b_{h}^{j},\mathsf{Var}_{\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $$

We now introduce the following lemma on $\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right>$, which is an empirical-transition version of Lemma [16](#Thmlemma16).

###### Lemma 9 .

Let $\delta\in(0,1)$. With probability at least $1-\delta$, the following condition holds for all $(h,i)\in[H]\times[n]$:

$$ $\displaystyle\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>\leq 3H\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]\right).$ (91) $$

###### Proof.

See Appendix [C.4.6](#A3.SS4.SSS6).
∎

Therefore, we can further achieve the following upper bound of $\mathcal{D}_{3}$ by applying Lemma [9](#Thmlemma9):

$$ $\displaystyle\mathcal{D}_{3}$ $\displaystyle\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\leq 9c_{\mathsf{b}}\sqrt{\frac{H\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]\right)$ $\displaystyle\overset{(\mathrm{i})}{\leq}9c_{\mathsf{b}}\sqrt{\frac{H\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\min\left\{\frac{1}{\sigma_{i}},H\right\}$ (92) $$

where (i) holds due to Lemma [8](#Thmlemma8).

###### Lemma 9 .

###### Proof.

#### C.2.3 Step 3: summing up the result

We combine the result of ([88](#A3.E88)), ([90](#A3.E90)), ([92](#A3.E92)), yielding

$$ $\displaystyle\widehat{V}_{i,h}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]\leq 3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}1+\mathcal{D}_{1}+\mathcal{D}_{2}+\mathcal{D}_{3}$ $\displaystyle\leq c_{\mathsf{b}}\sqrt{\frac{H\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\left(3H+15\min\left\{\frac{1}{\sigma_{i}},H\right\}\right)1$ $\displaystyle\quad+3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\max_{h\leq j\leq H}\left\lVert\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right\rVert_{\infty}1.$ $$

Moreover, Lemma [5](#Thmlemma5) implies that $\widehat{V}_{i,h}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]=\left|\widehat{V}_{i,h}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]\right|$, which indicates that

$$ $\displaystyle\max_{h\in[H]}\left\lVert\widehat{V}_{i,h}--\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]\right\rVert_{\infty}$ $\displaystyle\leq c_{\mathsf{b}}\sqrt{\frac{H\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\left(3H+15\min\left\{\frac{1}{\sigma_{i}},H\right\}\right)$ $\displaystyle\quad+3c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}\max_{h\leq j\leq H}\left\lVert\widehat{V}_{i,j+1}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right\rVert_{\infty}$ $\displaystyle\overset{\mathrm{(i)}}{\leq}18c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}+\frac{1}{2}\max_{h\in[H]}\left\lVert\widehat{V}_{i,h}-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]\right\rVert_{\infty}$ $\displaystyle\leq 36c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}$ $$

where (i) holds by taking $K\geq 12c_{\mathsf{b}}^{2}H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})$, and involving the basic facts that $\widehat{V}_{i,H+1}=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,H+1}^{\pi}\right]=0$. Eventually, we can achieve the following upper bound of term $B$:

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]\leq 36c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}1.$ (93) $$

### C.3 Controlling terms A and C

In this section, we derive an upper bound for the difference between the true value function and the estimated value function. We consider a more general case involving a given set of policies $\left\{\widehat{\pi}_{h}^{k}\right\}_{(h,k)\in[H]\times[K]}$, where either $\widehat{\pi}_{h}^{k}=\pi_{h}^{k}$ for all $(h,k)\in[H]\times[K]$, or $\widehat{\pi}_{h}^{k}=\tilde{\pi}_{i}^{\star}\times\pi_{-i,h}^{k}$ for all $(h,k)\in[H]\times[K]$. Additionally, we define a distribution over the set of policies $\zeta\coloneqq\{\zeta_{h}\}_{h\in[H]}$, with $\zeta_{h}:[H]\mapsto\Delta(\mathcal{S}\mapsto\prod_{i\in[n]}\Delta(\mathcal{A}_{i}))$, where $\zeta_{h}\left(\widehat{\pi}_{h}^{k}\right)=\alpha_{k}^{K}$ for all $(h,k)\in[H]\times[K]$.
Our objective is to derive an upper bound for
$\left|\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}(s)\right]\right|,$
where for all $s\in\mathcal{S}$, $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ is defined as

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}(s)\right]=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}(s)}[r_{i,h}^{k}(s,a_{i})]+\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}(s)}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h+1}\right]\right],$ $$

with $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,H+1}(s)\right]=0$. Here, $r_{i,h}^{k}(s,a_{i})$ represents the empirical estimation of $r_{i,h}^{\widehat{\pi}_{-i}^{k}}(s,a_{i})$, and $\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}$ denotes the empirical estimation of $P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}$ for all $(h,s,a_{i},k)\in[H]\times\mathcal{S}\times\mathcal{A}_{i}\times[K]$.
For notational clarity, we define the empirical reward vector $\overline{r}_{i,h}^{\widehat{\pi}^{k}}\in\mathbb{R}^{S}$, such that $\overline{r}_{i,h}^{\widehat{\pi}^{k}}(s)=\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}(s)}[r_{i,h}^{k}(s,a_{i})]$ for all $s\in\mathcal{S}$.

We first introduce the following two lemmas in terms of estimation error of transition model and reward function:

###### Lemma 10 .

Let $\delta\in(0,1)$ and consider any $(h,i,k)\in[H]\times[n]\times[K]$. With a probability of at least $1-\delta$, for any fixed value vector $V\in\mathbb{R}^{S}$, where $0\leq V(s)\leq H$ for all $s\in\mathcal{S}$, the following inequality holds:

$$ $\displaystyle\left|P_{i,h}^{\widehat{\pi}_{-i}^{k},V}V-\widehat{P}_{i,h}^{\widehat{\pi}_{-i}^{k},V}V\right|$ $\displaystyle\leq 2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}NHK}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{P_{h}^{\widehat{\pi}_{-i}^{k}}}(V)}+\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}NHK}{\delta}\right)}{N}1$ $\displaystyle\leq 3\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}NKH}{\delta}\right)}{N}}1,$ $$

where $\mathsf{Var}_{P_{h}^{\widehat{\pi}_{-i}^{k}}}(\cdot)$ is as defined in ([34](#A1.E34)).

###### Proof.

See Appendix [C.4.7](#A3.SS4.SSS7).
∎

###### Lemma 11 .

There exists a constant $c_{r}$ such that for any fixed pair $(h,i)\in[H]\times[n]$, with probability at least $1-\delta$, the following inequality holds:

$$ $\displaystyle\left|\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}-\sum_{k=1}^{K}\alpha_{k}^{K}\overline{r}_{i,h}^{\widehat{\pi}^{k}}\right|\leq c_{r}\sqrt{\frac{\log\left(\frac{KS}{\delta}\right)}{K}}1.$ $$

###### Proof.

See Appendix [C.4.8](#A3.SS4.SSS8).
∎

For clarity of presentation, we extend the definitions in ([32](#A1.E32)) and introduce additional notations related to transitions associated with the estimated value function. With a slight abuse of notation, we define the matrix notations $\widehat{P}_{i,h}^{\widehat{\pi}^{k},\widehat{V}}$ and $\widehat{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}$ as follows for all $(i,h,k)\in[n]\times[H]\times[K]$:

$$ $\displaystyle P_{i,h}^{\widehat{\pi}^{k},\overline{V}}:=P_{i,h}^{\widehat{\pi}_{-i}^{k},\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]}\quad\text{and}\quad P_{i,h,s,a_{i}}^{\widehat{\pi}^{k},\overline{V}}:=P_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k},\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}\left(P_{h,s,a_{i}}^{\widehat{\pi}^{k}_{-i}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right],$ $\displaystyle\widehat{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}:=\widehat{P}_{i,h}^{\widehat{\pi}_{-i}^{k},\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]}\quad\text{and}\quad\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}^{k},\overline{V}}:=\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k},\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]}=\mathrm{argmin}_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}_{\rho}\left(\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}^{k}_{-i}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right].$ $$

Additionally, we define the square matrices $\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\in\mathbb{R}^{S\times S}$ and $\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\in\mathbb{R}^{S\times S}$ as:

$$ $\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}:=\Pi_{h}^{\widehat{\pi}_{i}^{k}}\widehat{P}_{i,h}^{\widehat{\pi}_{-i}^{k},\overline{V}},\quad\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}:=\Pi_{h}^{\widehat{\pi}_{i}^{k}}P_{i,h}^{\widehat{\pi}_{-i}^{k},\overline{V}}.$ $$

At any time step $h\in[H]$, we have

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]$ $\displaystyle\overset{(\mathrm{i})}{=}$ $\displaystyle\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}+\sum_{k=1}^{K}\alpha_{k}^{K}\Pi_{h}^{\widehat{\pi}_{i,h}^{k}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h+1}\right]\right]$ $\displaystyle\quad-\sum_{k=1}^{K}\alpha_{k}^{K}\overline{r}_{i,h}^{\widehat{\pi}^{k}}-\sum_{k=1}^{K}\alpha_{k}^{K}\Pi_{h}^{\widehat{\pi}_{i,h}^{k}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(\widehat{P}_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h+1}\right]\right]$ $\displaystyle\overset{(\mathrm{ii})}{=}$ $\displaystyle\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]-\sum_{k=1}^{K}\alpha_{k}^{K}\overline{r}_{i,h}^{\widehat{\pi}^{k}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]$ (94) $$

where $\mathrm{(i)}$ holds by the robust Bellman equation in ([13](#S3.E13)) with matrix notation in ([32](#A1.E32)), $\mathrm{(ii)}$ arises from the definition in ([84](#A3.E84)). Moreover, through simple observation, we directly have $\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]\leq\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]$ for all $(h,k)\in[H]\times[K]$. Thus, we further control ([94](#A3.E94)) as

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]-\sum_{k=1}^{K}\alpha_{k}^{K}\overline{r}_{i,h}^{\widehat{\pi}^{k}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\bigg{[}\left(r_{i,h}^{\widehat{\pi}^{k}}-\overline{r}_{i,h}^{\widehat{\pi}^{k}}\right)+\left(\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}[V_{i,h+1}^{\pi}]-\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle\quad+\left(\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)\bigg{]}$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\left(\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]-\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle\quad+\underbrace{\sum_{k=1}^{K}\alpha_{k}^{K}\left[\left|r_{i,h}^{\widehat{\pi}^{k}}-\overline{r}_{i,h}^{\widehat{\pi}^{k}}\right|+\left|\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right|\right]}_{:=a_{i,h}^{\zeta}}.$ (95) $$

Applying ([95](#A3.E95)) recursively leads to

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]\leq\sum_{j=h}^{H}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},\overline{V}}\right)\right]a_{i,j}^{\zeta},$ (96) $$

where the inequality holds by adopting the following notations:

$$ $\displaystyle\left[\prod_{r=h}^{h-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},\overline{V}}\right)\right]=I,$ $\displaystyle\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},\overline{V}}\right)\right]=\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\right)\cdot\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h+1}^{\widehat{\pi}^{k},\overline{V}}\right)\cdots\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j-1}^{\widehat{\pi}^{k},\overline{V}}\right).$ $$

Next, similar to ([95](#A3.E95)), we can achieve that

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]$ $\displaystyle\overset{(\mathrm{i})}{=}$ $\displaystyle\sum_{k=1}^{K}\alpha_{k}^{K}\overline{r}_{i,h}^{\widehat{\pi}^{k}}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\Bigg{[}\left(\overline{r}_{i,h}^{\widehat{\pi}^{k}}-r_{i,h}^{\widehat{\pi}^{k}}\right)+\left(\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle\quad+\left(\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]\right)\Bigg{]}$ $\displaystyle\overset{(\mathrm{ii})}{\leq}$ $\displaystyle\sum_{k=1}^{K}\alpha_{k}^{K}\left(\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]\right)$ $\displaystyle\quad+\sum_{k=1}^{K}\alpha_{k}^{K}\left[\left|r_{i,h}^{\widehat{\pi}^{k}}-\overline{r}_{i,h}^{\widehat{\pi}^{k}}\right|+\left|\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right|\right],$ $$

where $(\mathrm{i})$ holds due to robust Bellman equation, and $(\mathrm{ii})$ holds due to the direct observation that $\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\leq\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]$. Then following the routine of achieving ([96](#A3.E96)), we can obtain that

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]\leq\sum_{j=h}^{H}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},V}\right)\right]a_{i,j}^{\zeta}.$ (97) $$

Summing up ([97](#A3.E97)) and ([96](#A3.E96)), one has

$$ $\displaystyle\left|\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]\right|$ $\displaystyle\leq\max\left\{\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right],\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]\right\}$ $\displaystyle\leq\max\left\{\sum_{j=h}^{H}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},\overline{V}}\right)\right]a_{i,j}^{\zeta},\sum_{j=h}^{H}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},V}\right)\right]a_{i,j}^{\zeta}.\right\},$ (98) $$

where the max operator is taken entry-wise for vectors. To continue, we apply Lemma [10](#Thmlemma10) and Lemma [11](#Thmlemma11), and we can obtain the following upper bound on $a_{i,j}^{\zeta}$ for all $(i,j)\in[n]\times[H]$:

$$ $\displaystyle a_{i,h}^{\zeta}$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\left[\left|r_{i,h}^{\widehat{\pi}^{k}}-\overline{r}_{i,h}^{\widehat{\pi}^{k}}\right|+\left|\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right|\right]$ $\displaystyle\leq 2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{h}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h+1}\right]\right)}$ $\displaystyle\quad+\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}1+c_{r}\sqrt{\frac{\log(\frac{KSnH}{\delta})}{K}}1,$ $$

holds with probability at least $1-\delta$.

###### Lemma 10 .

###### Proof.

###### Lemma 11 .

###### Proof.

#### C.3.1 Controlling the first term in ( 98 )

To simplify notation, let us introduce some additional symbols. Recall that $e_{s}$ represents the standard basis vector in $S$-dimensional space associated with the $s$-th component. We define

$$ $\displaystyle d_{h}^{h}=e_{s}\quad\text{and}\quad d_{h}^{j}=e_{s}^{\top}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},\overline{V}}\right)\right]\quad\text{for }j=h+1,\ldots,H.$ (99) $$

With these notations in place, for any $s\in\mathcal{S}$, we consider

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]=\left\langle e_{s},\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]\right\rangle=\sum_{j=h}^{H}\left\langle d_{h}^{j},a_{i,j}^{\zeta}\right\rangle.$ $$

Applying Lemma [10](#Thmlemma10), we obtain

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq$ $\displaystyle\sum_{j=h}^{H}\left\langle d_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right\rangle$ $\displaystyle\quad+\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{\log\left(\frac{KSnH}{\delta}\right)}{K}}$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log\left(\frac{KSnH}{\delta}\right)}{K}}$ $\displaystyle\quad+\sum_{j=h}^{H}\left\langle d_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right\rangle.$ (100) $$

By applying the triangle inequality, we can further decompose the term of interest as follows:

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log\left(\frac{KSnH}{\delta}\right)}{K}}$ $\displaystyle\quad+\underbrace{\sum_{j=h}^{H}\left\langle d_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right\rangle}_{\mathcal{B}_{1}}$ $\displaystyle+\underbrace{\sum_{j=h}^{H}2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNH}{\delta}\right)}{N}}\left\langle d_{h}^{j},\sqrt{\left|\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right|}\right\rangle}_{\mathcal{B}_{2}}.$ (101) $$

We then analyze the bounds for the terms $\mathcal{B}_{1}$ and $\mathcal{B}_{2}$ separately.

##### Controlling ℬ 1 subscript ℬ 1 \mathcal{B}_{1} .

First, we introduce the following lemma and corresponding inequality to establish control over the term $\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}$:

###### Lemma 12 .

For any transition kernels $P_{1},\ldots,P_{m}\in\mathbb{R}^{S}$, and any weight $a_{1},\ldots,a_{m}\in[0,1]$ with $a_{1}+\ldots+a_{m}=1$, one has

$$ $\displaystyle\sum_{i=1}^{m}a_{i}\sqrt{\mathsf{Var}_{P_{i}}(V)}\leq\sqrt{\mathsf{Var}_{\sum_{i=1}^{m}a_{i}P_{i}}(V)},$ $$

where $V$ denote any fixed value vector $V\in\mathbb{R}^{S}$ with $0\leq V(s)\leq H$ for all $s\in\mathcal{S}$.

###### Proof.

Initially, since $f(x)=\sqrt{x}$ is a concave function, we have

$$ $\displaystyle\sum_{i=1}^{m}a_{i}\sqrt{\mathsf{Var}_{P_{i}}(V)}=\sqrt{\sum_{i=1}^{m}a_{i}\mathsf{Var}_{P_{i}}(V)}.$ $$

Moreover, according to the definition of variance in ([33](#A1.E33)), we obtain that

$$ $\displaystyle\sum_{i=1}^{m}a_{i}\mathsf{Var}_{P_{i}}(V)$ $\displaystyle=\sum_{i=1}^{m}a_{i}\left(\mathbb{E}_{P_{i}}\left(V\circ V\right)-\left(\mathbb{E}_{P_{i}}V\circ\mathbb{E}_{P_{i}}V\right)\right)\leq\sum_{i=1}^{m}a_{i}\mathbb{E}_{P_{i}}\left(V\circ V\right)-\left(\sum_{i=1}^{m}a_{i}\mathbb{E}_{P_{i}}V\right)\circ\left(\sum_{i=1}^{m}a_{i}\mathbb{E}_{P_{i}}V\right),$ $$

where the last inequality holds due to the elementary fact that $f(x)=x^{2}$ is a convex function. Therefore, we have proven the result of the lemma.
∎

With Lemma [12](#Thmlemma12), we can further control $\mathcal{B}_{1}$ with

$$ $\displaystyle\mathcal{B}_{1}$ $\displaystyle=\sum_{j=h}^{H}\left<d_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right>$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sum_{j=h}^{H}\left<d_{h}^{j},\sqrt{\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right>$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{H\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>}.$ (102) $$

The last inequality holds due to Cauchy-Schwartz inequality. To further achieve the upper bound of $\mathcal{B}_{1}$, we introduce the following lemma of $\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$:

###### Lemma 13 .

Consider any $\delta\in(0,1)$. With probability at least $1-\delta$, one has

$$ $\displaystyle\forall(h,i)\in[H]\times[n]:\quad$ $\displaystyle\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle\leq$ $\displaystyle 3H\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]\right)\left(1+2H\sqrt{\frac{\log(\frac{18S\sum_{i=1}^{n}A_{i}nKNH}{\delta})}{N}}\right).$ (103) $$

###### Proof.

See Appendix [C.4.9](#A3.SS4.SSS9).
∎

###### Lemma 14 .

For all $(i,h)\in[n]\times[H]$, the estimated robust value function $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]$ satisfies the following inequality:

$$ $\displaystyle\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ $$

###### Proof.

The proof of Lemma [14](#Thmlemma14) closely parallels that of Lemma [7](#Thmlemma7). Therefore, we omit the details here for brevity and clarity.
∎

Apply Lemma [13](#Thmlemma13) to ([102](#A3.E102)), we arrive at

$$ $\displaystyle\mathcal{B}_{1}$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{H\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>}$ $\displaystyle\leq\sqrt{3H^{2}\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]\right)}$ $\displaystyle\cdot 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}\cdot\left(1+2H\sqrt{\frac{\log(\frac{18S\sum_{i=1}^{n}A_{i}nKNH}{\delta})}{N}}\right)}$ $\displaystyle\overset{(\mathsf{i})}{\leq}$ $\displaystyle 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nKNnH}{\delta}\right)}{N}}\sqrt{3H^{2}\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}\left(1+2H\sqrt{\frac{\log(\frac{18S\sum_{i=1}^{n}A_{i}nKNH}{\delta})}{N}}\right)}$ $\displaystyle\leq 6\sqrt{\frac{H^{2}\min\{1/\sigma_{i},H\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}},$ (104) $$

where (i) holds by applying Lemma [7](#Thmlemma7) and Lemma [14](#Thmlemma14), and the final inequality follows by taking $N\geq 4H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KnNH}{\delta}\right)$.

###### Lemma 12 .

###### Proof.

###### Lemma 13 .

###### Proof.

###### Lemma 14 .

###### Proof.

##### Controlling ℬ 2 subscript ℬ 2 \mathcal{B}_{2} .

Initially, with similar analysis as Lemma [8](#Thmlemma8), we have the following lemma:

###### Lemma 15 .

For transition kernel $P^{\prime}\in\mathbb{R}^{S}$ and any $\widetilde{P}\in\mathbb{R}^{S}$ such that $\widetilde{P}\in\mathcal{U}^{\sigma_{i}}\left(P^{\prime}\right)$, the following bounds are established for all $(i,h)\in[n]\times[H]$:

$$ $\displaystyle\left|\mathsf{Var}_{P^{\prime}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]\right)-\mathsf{Var}_{\widetilde{P}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]\right)\right|\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ $$

With Lemma [15](#Thmlemma15), we observe that

$$ $\displaystyle\left|\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right|$ $\displaystyle\overset{\mathrm{(i)}}{=}\left|\Pi_{j}^{\pi_{i}}\left(\mathsf{Var}_{P^{\widehat{\pi}^{k}_{-i}}_{j}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{P_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right)\right|$ $\displaystyle\overset{\mathrm{(ii)}}{\leq}\left\|\mathsf{Var}_{P^{\widehat{\pi}^{k}_{-i}}_{j}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{P_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right\|_{\infty}1$ $\displaystyle\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}1,$ (105) $$

where (i) and (ii) follows from the definition of matrix notations $\Pi_{j}^{\pi}$ (cf [A.1](#A1.SS1.SSS0.Px1)) and $\underline{P}_{j}^{\widehat{\pi}^{k}},\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}$ (cf [A.1](#A1.SS1.SSS0.Px3)), and the last inequality holds by applying Lemma [15](#Thmlemma15) with $P^{\prime}=P^{\pi_{-i}^{k}}_{j,s,a_{i}},\widetilde{P}=P_{i,j,s,a_{i}}^{\widehat{\pi}^{k},\overline{V}}$ for all $(s,a_{i})\in\mathcal{S}\times\mathcal{A}_{i}$.

Plugging back ([105](#A3.E105)) to ([101](#A3.E101)), it can be verified that

$$ $\displaystyle\mathcal{B}_{2}$ $\displaystyle=\sum_{j=h}^{H}2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\left<d_{h}^{j},\sqrt{\left|\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right|}\right>$ $\displaystyle\leq\sum_{j=h}^{H}2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\left<d_{h}^{j},\sqrt{\min\left\{\frac{1}{\sigma_{i}},H\right\}}1\right>$ $\displaystyle\leq 2\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}$ (106) $$

Consequently, combining ([104](#A3.E104)) and ([106](#A3.E106)), ([101](#A3.E101)) can be bounded by

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}+\mathcal{B}_{1}+\mathcal{B}_{2}$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}+8\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}$ $\displaystyle\leq c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}+12\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}},$ (107) $$

where the last inequality holds by taking $N\geq 4H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)$.

###### Lemma 15 .

#### C.3.2 Controlling the second term in ( 98 )

To do so, similar to ([99](#A3.E99)), we define

$$ $\displaystyle w_{h}^{h}=e_{s}\quad\text{and}\quad w_{h}^{j}=e_{s}^{\top}\left[\prod_{r=h}^{j-1}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,r}^{\widehat{\pi}^{k},V}\right)\right]\quad\forall j=h+1,\cdots,H.$ (108) $$

With the above notations in mind, following the routine of ([100](#A3.E100)) gives: for any $s\in\mathcal{S}$,

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ $\displaystyle\leq\sum_{j=h}^{H}\bigg{<}w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}$ $\displaystyle\quad+\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}1+c_{r}\sqrt{\frac{\log(\frac{KSnH}{\delta})}{K}}1\bigg{>}$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}$ $\displaystyle\quad+\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right>.$ (109) $$

Furthermore, following the routine established in ([101](#A3.E101)), we can decompose the expression as follows:

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}$ $\displaystyle\quad+\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)}\right>$ $\displaystyle\overset{(\mathrm{i})}{\leq}\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}$ $\displaystyle+\underbrace{\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)}\right>}_{\mathcal{B}_{3}}$ $\displaystyle\quad+\underbrace{\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\left|\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right|}\right>}_{\mathcal{B}_{4}}$ $\displaystyle\quad+\underbrace{\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)}\right>}_{\mathcal{B}_{5}}$ (110) $$

where (i) holds due to the triangle inequality and the fundamental inequality $\sqrt{\mathsf{Var}_{P}(V+V^{\prime})}\leq\sqrt{\mathsf{Var}_{P}(V)}+\sqrt{\mathsf{Var}_{P}(V^{\prime})}$ for any transition kernel $P\in\mathbb{R}^{S}$ and vectors $V,V^{\prime}\in\mathbb{R}^{S}$.

Next, we will control the three main terms $\mathcal{B}_{3},\mathcal{B}_{4},\mathcal{B}_{5}$ in ([110](#A3.E110)) separately as outlined below:

##### Controlling ℬ 3 subscript ℬ 3 \mathcal{B}_{3} .

Initially, we apply Lemma [12](#Thmlemma12), and we can obtain the following upper bound of $\mathcal{B}_{3}$:

$$ $\displaystyle\mathcal{B}_{3}$ $\displaystyle=\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)}\right>$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sum_{j=h}^{H}\left<w_{h}^{j},\sqrt{\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)}\right>.$ $$

We further apply Cauchy-Schwartz inequality, and we can obtain that

$$ $\displaystyle\mathcal{B}_{3}$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sum_{j=h}^{H}\left<w_{h}^{j},\sqrt{\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)}\right>$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{H\sum_{j=h}^{H}\left<w_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right>}$ $$

In addition, to further bound the term of interest, we introduce the following lemma and inequalities for $\sum_{j=h}^{H}\left<w_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right>$.

###### Lemma 16 .

For any joint policy $\pi$, we have for all $(h,i)\in[H]\times[n]$:

$$ $\displaystyle\sum_{j=h}^{H}\left<w_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right>\leq 3H\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]\right).$ (111) $$

###### Proof.

See Appendix [C.4.10](#A3.SS4.SSS10).
∎

###### Lemma 17 .

For all $(i,h)\in[n]\times[H]$, the estimated robust value function $\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}\right]$ satisfies the following inequality:

$$ $\displaystyle\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ $$

###### Proof.

The proof of Lemma [17](#Thmlemma17) closely parallels that of Lemma [7](#Thmlemma7). Therefore, we omit the details here for brevity and clarity.
∎

Then applying Lemma [16](#Thmlemma16) and Lemma [17](#Thmlemma17) yields

$$ $\displaystyle\mathcal{B}_{3}$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{H\sum_{j=h}^{H}\left<w_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right>}$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{3H^{2}\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi,\sigma_{i}}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi,\sigma_{i}}(s)\right)\right]}$ $\displaystyle\leq 4\sqrt{\frac{H^{2}\min\left\{1/\sigma_{i},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}},$ (112) $$

where the last inequality follows from Lemma [17](#Thmlemma17).

###### Lemma 16 .

###### Proof.

###### Lemma 17 .

###### Proof.

##### Controlling ℬ 4 subscript ℬ 4 \mathcal{B}_{4} and ℬ 5 subscript ℬ 5 \mathcal{B}_{5}

With similar analysis as Lemma [8](#Thmlemma8), we have the following lemma:

###### Lemma 18 .

For any joint policy $\pi$, transition kernel $P^{\prime}\in\mathbb{R}^{S}$, and any $\widetilde{P}\in\mathbb{R}^{S}$ such that $\widetilde{P}\in\mathcal{U}^{\sigma_{i}}\left(P^{\prime}\right)$, the following bounds are established for all $(i,h)\in[n]\times[H]$:

$$ $\displaystyle\left|\mathsf{Var}_{P^{\prime}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}\right]\right)-\mathsf{Var}_{\widetilde{P}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}\right]\right)\right|\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ $$

We apply Lemma [18](#Thmlemma18), and we can directly obtain the following upper bound of $\mathcal{B}_{4}$:

$$ $\displaystyle\mathcal{B}_{4}$ $\displaystyle=\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\left|\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)-\mathsf{Var}_{\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right|}\right>$ $\displaystyle\leq\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\min\left\{\frac{1}{\sigma_{i}},H\right\}}1\right>$ $\displaystyle=2\sqrt{\frac{H^{2}\min\{1/\sigma_{i},H\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}$ (113) $$

Then the remainder of the proof shall focus on $\mathcal{B}_{5}$. Recalling the definition in ([110](#A3.E110)), one has

$$ $\displaystyle\mathcal{B}_{5}$ $\displaystyle=\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)}\right>$ $\displaystyle\leq\sum_{j=h}^{H}\left<w_{h}^{j},2\sum_{k=1}^{K}\alpha_{k}^{K}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sqrt{\left\lVert\mathsf{Var}_{\underline{P}_{j}^{\widehat{\pi}^{k}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right\rVert_{\infty}}1\right>$ $\displaystyle\leq 2\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\sum_{j=h}^{H}\left<w_{h}^{j},\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right\rVert_{\infty}1\right>$ $\displaystyle\leq 2\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\max_{h\leq j\leq H}\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right\rVert_{\infty}.$ (114) $$

Summing up ([112](#A3.E112)), ([113](#A3.E113)), and ([114](#A3.E114)) and inserting back to ([110](#A3.E110)), we conclude

$$ $\displaystyle\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}(s)\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log\left(\frac{KSnH}{\delta}\right)}{K}}+\mathcal{B}_{3}+\mathcal{B}_{4}+\mathcal{B}_{5}$ $\displaystyle\leq\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+c_{r}\sqrt{\frac{H^{2}\log\left(\frac{KSnH}{\delta}\right)}{K}}+6\sqrt{\frac{H^{2}\min\{1/\sigma_{i},H\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}$ $\displaystyle\quad+2\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\max_{h\leq j\leq H}\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right\rVert_{\infty}$ (115) $$

###### Lemma 18 .

#### C.3.3 Summing up the results: upper bound for term A and C

Inserting ([107](#A3.E107)) and ([115](#A3.E115)) back into ([98](#A3.E98)), we observe that

$$ $\displaystyle\left|\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]\right|$ $\displaystyle\leq\max\left\{\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right],\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]\right\}$ $\displaystyle\leq\max\Bigg{\{}c_{r}\sqrt{\frac{H^{2}\log\left(\frac{KSnH}{\delta}\right)}{K}}1+12\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}1,\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}1$ $\displaystyle\quad+2\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\max_{h\leq j\leq H}\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right\rVert_{\infty}1+c_{r}\sqrt{\frac{H^{2}\log\left(\frac{KSnH}{\delta}\right)}{K}}1$ $\displaystyle\quad+6\sqrt{\frac{H^{2}\min\{\frac{1}{\sigma_{i}},H\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}1\Bigg{\}},$ $$

which indicates that

$$ $\displaystyle\max_{h\in[H]}\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[V^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}^{\pi}_{i,h}\right]\right\rVert_{\infty}$ (116) $\displaystyle\leq c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}+12\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}+\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}$ $\displaystyle\quad+2\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}\max_{h\in[H]}\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi}\right]\right\rVert_{\infty}$ $\displaystyle\overset{\mathrm{(i)}}{\leq}c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}+12\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}$ $\displaystyle\quad+\frac{H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}+\frac{1}{2}\max_{h\in[H]}\left\lVert\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]-\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}\right]\right\rVert_{\infty}$ $\displaystyle\leq 2c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}+24\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}+\frac{2H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}$ (117) $$

where (i) holds when $N\geq 4H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}NnH}{\delta}\right)$, and invoking the basic fact that $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,H+1}^{\pi}\right]=\mathbb{E}_{\pi\sim\zeta}\left[V_{i,H+1}^{\pi}\right]=0.$ With ([117](#A3.E117)), we can achieve the following upper bound on term $A$ and term $C$:

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V^{\star,\pi_{-i}}_{i,h}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\widetilde{\pi}_{i}^{\star},\pi_{-i}}\right]$ $\displaystyle\leq 2c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}1+24\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}1+\frac{2H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}1$ (118) $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,h}^{\pi}\right]$ $\displaystyle\leq 2c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}1+24\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}1+\frac{2H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}1$ (119) $$

#### C.3.4 Summing up the results

Summing up the results in ([118](#A3.E118)), ([93](#A3.E93)), ([119](#A3.E119)), we can achieve the upper bound of our target:

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\star,\pi_{-i}}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\pi}\right]$ $\displaystyle\leq 36c_{\mathsf{b}}\sqrt{\frac{H^{3}\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{K}}1+4c_{r}\sqrt{\frac{H^{2}\log(\frac{KSnH}{\delta})}{K}}1$ $\displaystyle\quad+\frac{4H\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}1+48\sqrt{\frac{H^{2}\min\left\{\frac{1}{\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)}{N}}1.$ $$

Therefore, there exists a constant $C$, such that when $N$ and $K$ satisfies:

$$ $\displaystyle N\geq CH^{2}\min\left\{\frac{1}{\min_{1\leq i\leq n}\sigma_{i}},H\right\}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}KNnH}{\delta}\right)\frac{1}{\epsilon^{2}},\quad K\geq CH^{3}\log^{3}\left(\frac{KS\sum_{i=1}^{n}A_{i}H}{\delta}\right)\frac{1}{\epsilon^{2}}$ $$

we achieve $\max_{i\in[n]}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\star,\pi_{-i}}\right]-\mathbb{E}_{\pi\sim\widehat{\xi}}\left[V_{i,1}^{\pi}\right]\leq\epsilon\cdot 1$ with probability at least $1-\delta$.
Therefore, the total number of samples we need is at least

$$ $\displaystyle N_{\mathsf{all}}=HS\sum_{i=1}^{n}KN=\tilde{\mathcal{O}}\left(\frac{S\max_{1\leq i\leq n}A_{i}H^{6}}{\epsilon^{4}}\min\left\{\frac{1}{\min_{1\leq i\leq n}\sigma_{i}},H\right\}\right).$ $$

Thus, we finish the proof of Theorem [2](#Thmtheorem2).

### C.4 Proof of auxiliary lemmas

#### C.4.1 Proof of Lemma 4

Before proving Lemma [4](#Thmlemma4), we first introduce the following lemma regarding the properties of the learning rate.

###### Lemma 19 ( Li et al., ( 2023 , Lemma 1) ) .

For any $k\geq 1$, one has

$$ $\alpha_{1}=1,\qquad\sum_{i=1}^{k}\alpha_{i}^{k}=1,\qquad\max_{1\leq i\leq k}\alpha_{i}^{k}\leq\frac{2c_{\alpha}\log K}{k}.$ (120a) In addition, if $k\geq c_{\alpha}\log K+1$ and $c_{\alpha}\geq 24$ , then one has $\max_{1\leq i\leq k/2}\alpha_{i}^{k}\leq\frac{1}{K^{6}}.$ (120b) $$

We will now prove the lemma with induction argument. Initially, the base step $H+1$ trivially holds true, since we have

$$ $\displaystyle\widehat{V}_{i,H+1}=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,H+1}^{\star,\pi_{-i}}\right]=0.$ $$

Next, we assume that the lemma holds for step $h+1$, namely

$$ $\displaystyle\widehat{V}_{i,h+1}\geq\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\star,\pi_{-i}}\right]$ $$

and attempt to justify the validity of Lemma [4](#Thmlemma4) for step $h$.
Let $l_{k}$ denote $l_{k}=-q_{i,h}^{k}(s,\cdot),\forall k\geq 1$, then the update rule of Algorithm [2](#alg2) can be viewed as the FTRL algorithm applied to the loss vectors $\{l_{k}\}_{k\in[K]}$. According to the definition of $\{\eta_{k}\}_{k\in[K]}$ and $\{\alpha_{k}\}_{k\in[K]}$, we have

$$ $\displaystyle\bigg{(}\frac{\eta_{k}}{\eta_{k+1}}\bigg{)}^{2}$ $\displaystyle=\frac{\alpha_{k}}{\alpha_{k-1}}=\frac{k-2+c_{\alpha}\log K}{k-1+c_{\alpha}\log K}\geq\frac{k-1}{k-1+c_{\alpha}\log K}=1-\alpha_{k}>(1-\alpha_{k})^{2}.$ (121) $$

This property ([121](#A3.E121)) permits us to invoke Theorem [3](#Thmtheorem3) to obtain

$$ $\displaystyle\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}q_{i,h}^{k}(s,a_{i})-\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{i,h}^{k},q_{i,h}^{k}(s,\cdot)\right>$ $\displaystyle=\max_{a_{i}\in\mathcal{A}_{i}}\left\{\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{i,h}^{k}(s),l_{k}\right>-\sum_{k=1}^{K}\alpha_{k}^{K}l_{k}(a_{i})\right\}$ $\displaystyle\leq\frac{5}{3}\sum_{k=2}^{K}\alpha_{k}^{K}\frac{\eta_{k}\alpha_{k}}{1-\alpha_{k}}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}+\frac{\log A_{i}}{\eta_{K+1}}+\tau_{i,h}$ $$

where $\tau_{i,h}$ is defined as

$$ $\displaystyle\tau_{i,h}$ $\displaystyle:=\frac{5}{3}\alpha_{1}^{K}\eta_{2}\big{\|}q_{i,h}^{1}\big{\|}_{\infty}^{2}+\left\{3\sum_{k=2}^{K}\alpha_{k}^{K}\frac{\eta_{k}^{2}\alpha_{k}^{2}}{(1-\alpha_{k})^{2}}\big{\|}q_{i,h}^{k}\big{\|}_{\infty}^{3}\mathbf{1}\bigg{(}\frac{\eta_{k}\alpha_{k}}{1-\alpha_{k}}\big{\|}q_{i,h}^{k}\big{\|}_{\infty}>\frac{1}{3}\bigg{)}\right\}+3\alpha_{1}^{K}\eta_{2}^{2}\big{\|}q_{i,h}^{1}\big{\|}_{\infty}^{3}.$ (122) $$

According to the definition of $\{\alpha_{k}\}_{k=1}^{K}$ and $\{\eta_{k}\}_{k=1}^{K}$, we have the following fact:

$$ $\displaystyle 1-\alpha_{k}$ $\displaystyle=1-\frac{c_{\alpha}\log K}{k-1+c_{\alpha}\log K}\geq\begin{cases}1-\frac{c_{\alpha}\log K}{1+c_{\alpha}\log K}=\frac{1}{1+c_{\alpha}\log K}\geq\frac{1}{2c_{\alpha}\log K},&\text{if }k\geq 2,\\ 1-\frac{c_{\alpha}\log K}{K/2+c_{\alpha}\log K}=\frac{K}{K+2c_{\alpha}\log K}\geq\frac{1}{2},&\text{if }k\geq K/2+1,\end{cases}$ (123a) $\displaystyle\eta_{k}\alpha_{k}$ $\displaystyle=\sqrt{\frac{\log K}{\alpha_{k-1}H}}\cdot\alpha_{k}\leq\sqrt{\frac{\log K}{\alpha_{k}H}}\cdot\alpha_{k}=\sqrt{\frac{\alpha_{k}\log K}{H}}\leq\sqrt{\frac{2c_{\alpha}\log^{2}K}{kH}}.$ (123b) $$

Therefore, we can re-control $\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}q_{i,h}^{k}(s,a_{i})-\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{i,h}^{k},q_{i,h}^{k}(s,\cdot)\right>$ with

$$ $\displaystyle\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}q_{i,h}^{k}(s,a_{i})-\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{i,h}^{k},q_{i,h}^{k}(s,\cdot)\right>$ (124) $\displaystyle\leq\frac{5}{3}\sum_{k=2}^{K}\alpha_{k}^{K}\frac{\eta_{k}\alpha_{k}}{1-\alpha_{k}}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}+\frac{\log A_{i}}{\eta_{K+1}}+\tau_{i,h}$ $\displaystyle\overset{(\mathrm{i})}{\leq}\underbrace{\frac{5}{3}\sum_{k=2}^{K/2}\frac{\big{(}2c_{\alpha}\big{)}^{1.5}\log^{2}K}{\sqrt{kH}}\alpha_{k}^{K}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}}_{\mathcal{C}_{1}}$ $\displaystyle\quad+\frac{20}{3}\sum_{k=K/2+1}^{K}\alpha_{k}^{K}\sqrt{\frac{c_{\alpha}\log^{2}K}{KH}}\,\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}+\underbrace{\frac{\log A_{i}}{\eta_{K+1}}}_{\mathcal{C}_{2}}+\underbrace{\tau_{i,h}}_{\mathcal{C}_{3}},$ (125) $$

Now we separately control the four terms $\mathcal{C}_{1},\mathcal{C}_{2},\mathcal{C}_{3}$ in ([125](#A3.E125)).

- •
For term $\mathcal{C}_{1}$, we have
$\displaystyle\sum_{k=2}^{K/2}\frac{\alpha_{k}^{K}\log^{2}K}{\sqrt{kH}}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}$
$\displaystyle\leq\sum_{k=2}^{K/2}\frac{\log^{2}K}{K^{6}\sqrt{kH}}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}$
$\displaystyle\leq\sum_{k=2}^{K/2}\frac{\log^{2}K}{K^{6}\sqrt{kH}}\big{\|}q_{i,h}^{k}(s,\cdot)\big{\|}_{\infty}^{2}\leq\frac{H^{3/2}\log^{2}K}{K^{6}}\sum_{k=2}^{K/2}\frac{1}{\sqrt{k}}$
$\displaystyle\leq\frac{2H^{3/2}\log^{2}K}{K^{6}}\cdot\sqrt{K/2}\leq\frac{2H^{3/2}\log^{2}K}{K^{5}},$
(126)
where the third inequality holds due to the elementary bound $\big{\|}q_{i,h}^{k}(s,\cdot)\big{\|}_{\infty}\leq H$.
- •
For term $\mathcal{C}_{2}$, we have
$\displaystyle\frac{\log A_{i}}{\eta_{K+1}}$
$\displaystyle=\log A_{i}\sqrt{\frac{\alpha_{K}H}{\log K}}\leq\sqrt{\frac{2c_{\alpha}H\log^{2}A_{i}}{K}},$
(127)
where the first equality holds due to the definition of $\eta_{K+1}$.
- •
For term $\mathcal{C}_{3}$, we initially have
$\displaystyle\frac{\eta_{k}\alpha_{k}}{1-\alpha_{k}}\big{\|}q_{i,h}^{k}\big{\|}_{\infty}$
$\displaystyle\leq\frac{\sqrt{\frac{2c_{\alpha}\log^{2}K}{kH}}}{\frac{1}{2c_{\alpha}\log K}}\cdot H=\sqrt{\frac{8c_{\alpha}^{3}H\log^{4}K}{k}}.$
(128)
Clearly, the right-hand side of ([128](#A3.E128)) is upper bounded by $1/3$ for all $k$ obeying $k\geq c_{9}H^{2}\log^{4}\frac{K}{\delta}$ for some large enough constant $c_{9}>0$.
Consequently, one can derive
$\displaystyle\tau_{i,h}$
$\displaystyle=\frac{5}{3}\alpha_{1}^{K}\eta_{2}\big{\|}q_{i,h}^{1}\big{\|}_{\infty}^{2}+\left\{3\sum_{k=2}^{K}\alpha_{k}^{K}\frac{\eta_{k}^{2}\alpha_{k}^{2}}{(1-\alpha_{k})^{2}}\big{\|}q_{i,h}^{k}\big{\|}_{\infty}^{3}\mathbf{1}\bigg{(}\frac{\eta_{k}\alpha_{k}}{1-\alpha_{k}}\big{\|}q_{i,h}^{k}\big{\|}_{\infty}>\frac{1}{3}\bigg{)}\right\}+3\alpha_{1}^{K}\eta_{2}^{2}\big{\|}q_{i,h}^{1}\big{\|}_{\infty}^{3}$
$\displaystyle\leq\frac{5}{3K^{6}}\sqrt{\frac{\log K}{H}}\big{\|}q_{i,h}^{1}\big{\|}_{\infty}^{2}+\frac{\big{(}2c_{\alpha}\log K\big{)}^{2}}{K^{6}}\left\{3\sum_{k=2}^{c_{9}H^{2}\log^{4}\frac{K}{\delta}}\eta_{k}^{2}\alpha_{k}^{2}\big{\|}q_{i,h}^{k}\big{\|}_{\infty}^{3}\right\}+\frac{3}{K^{6}}\frac{\log K}{H}\big{\|}q_{i,h}^{1}\big{\|}_{\infty}^{3}$
$\displaystyle\leq\frac{24c_{\alpha}^{3}\log^{4}K}{K^{6}H}\left\{\sum_{k=1}^{K}\frac{1}{k}H^{3}\right\}$
$\displaystyle\leq\frac{24c_{\alpha}^{3}H^{3}\log^{5}K}{K^{6}}\leq\frac{1}{K^{4}},$
(129)
where the second line comes from ([123](#A3.E123)) and the fact that $K/2>c_{9}H\log^{4}\frac{K}{\delta}$ .

Combining previous three items, we can obtain that

$$ $\displaystyle\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}q_{i,h}^{k}(s,a_{i})-\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{i,h}^{k},q_{i,h}^{k}(s,\cdot)\right>$ $\displaystyle\leq\frac{5(2c_{\alpha})^{1.5}}{3}\cdot\frac{2H^{3/2}\log^{2}K}{K^{5}}+\frac{20}{3}\sqrt{\frac{c_{\alpha}\log^{2}K}{KH}}\sum_{k=K/2+1}^{K}\alpha_{k}^{K}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}+\sqrt{\frac{2c_{\alpha}H\log^{2}A_{i}}{K}}+\frac{1}{K^{4}}$ $\displaystyle\leq 10\sqrt{\frac{c_{\alpha}\log^{3}(KA_{i})}{KH}}\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}+2\sqrt{\frac{c_{\alpha}H\log^{3}(KA_{i})}{K}},$ (130) $$

According to the definition of $q_{i,h}^{k}(s,a_{i})$ in the update rule of Algorithm [2](#alg2), we have

$$ $\displaystyle\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]-\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]$ $\displaystyle=\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}q_{i,h}^{k}(s,a_{i})-\sum_{k=1}^{K}\alpha_{k}^{K}\left<\pi_{i,h}^{k},q_{i,h}^{k}(s,\cdot)\right>$ $\displaystyle\leq 10\sqrt{\frac{c_{\alpha}\log^{3}(KA_{i})}{KH}}\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{\pi_{i,h}^{k}(s)}\Big{(}q_{i,h}^{k}(s,\cdot)\Big{)}+2\sqrt{\frac{c_{\alpha}H\log^{3}(KA_{i})}{K}}=\beta_{i,h}(s)$ $$

Moreover, according to the induction hypothesis, we have for all $s\in\mathcal{S}$

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\star,\pi_{-i}}_{i,h}(s)\right]$ $\displaystyle=\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P\mathbb{E}_{\pi\sim\widehat{\xi}}}\left[\overline{V}_{i,h+1}^{\star,\pi_{-i}}\right]\right]$ $\displaystyle\leq\max_{a_{i}\in\mathcal{A}_{i}}\sum_{k=1}^{K}\alpha_{k}^{K}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}(s)$ $\displaystyle\leq\widehat{V}_{i,h}(s).$ $$

Thus, we finished the proof of the lemma.

###### Lemma 19 ( Li et al., ( 2023 , Lemma 1) ) .

#### C.4.2 Proof of Lemma 5

We will prove the lemma with induction argument. Initially, the base step $H+1$ trivially holds true, since we have

$$ $\displaystyle\widehat{V}_{i,H+1}=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,H+1}\right]=0.$ $$

Next, we assume that the lemma holds for step $h+1$, namely

$$ $\displaystyle\widehat{V}_{i,h+1}\geq\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h+1}\right].$ $$

According to the definition of $\widehat{V}_{i,h}$ and $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}\right]$, we have

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right]$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}(s).$ $$

Since, we also trivially have $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}(s)\right]\leq H-h+1$, we can deduce that for all $s\in\mathcal{S}$

$$ $\displaystyle\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}^{\pi}_{i,h}(s)\right]$ $\displaystyle\leq\min\left\{\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})+\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}(P_{i,h,s,a_{i}}^{k})}\mathcal{P}\widehat{V}_{i,h+1}\right]+\beta_{i,h}(s),H-h+1\right\}=\widehat{V}_{i,h}(s).$ $$

Thus, we finished the proof of the lemma.

#### C.4.3 Proof of Lemma 6

Recall that for all $s\in\mathcal{S}$, bonus term $\beta_{i,h}(s)$ is defined as

$$ $\displaystyle\beta_{i,h}(s)=c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\sum_{k=1}^{K}\alpha_{k}^{K}\left\{\mathsf{Var}_{\pi_{i,h}^{k}(\cdot\mid s)}\left(q_{i,h}^{k}(s,\cdot)\right)+H\right\}.$ (131) $$

For any $k\in[K]$, we have the following inequality for $\mathsf{Var}_{\pi_{i,h}^{k}(\cdot\mid s)}\left(q_{i,h}^{k}(s,\cdot)\right)$:

$$ $\displaystyle\mathsf{Var}_{\pi_{i,h}^{k}(\cdot\mid s)}\left(q_{i,h}^{k}(s,\cdot)\right)$ $\displaystyle\leq 2\mathsf{Var}_{\pi_{i,h}^{k}(\cdot\mid s)}\left(r_{i,h}^{k}(s,\cdot)\right)+2\mathsf{Var}_{\pi_{i,h}^{k}(\cdot\mid s)}\left(\sum_{s^{\prime}}\widehat{P}_{i,h}^{\pi_{-i}^{k},\widehat{V}}(s^{\prime}\mid s,\cdot)\widehat{V}_{i,h+1}(s^{\prime})\right)$ $\displaystyle\overset{\mathsf{(i)}}{\leq}2+2\left[\sum_{a_{i}\in\mathcal{A}_{i}}\pi_{i,h}^{k}(a_{i}\mid s)\widehat{P}_{i,h}^{\pi_{-i}^{k},\widehat{V}}(\cdot\mid s,a_{i})\left(\widehat{V}_{i,h+1}\circ\widehat{V}_{i,h+1}\right)-\left(\sum_{a_{i}\in\mathcal{A}_{i}}\pi_{i,h}^{k}(a_{i}\mid s)\widehat{P}_{i,h}^{\pi_{-i}^{k},\widehat{V}}(\cdot\mid s,a_{i})\widehat{V}_{i,h+1}\right)^{2}\right]$ $\displaystyle=2+2\left<e_{s},\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\widehat{V}_{i,h+1}\right>.$ (132) $$

where $e_{s}$ denotes an $S$-dimensional standard basis supported on the $s$-th element, and (i) holds due to the elementary fact that $\left|r_{i,h}^{k}(s,a_{i})\right|\leq 1$ and $\left|\widehat{P}_{i,h}^{\pi_{-i}^{k},\widehat{V}}\left(s^{\prime}\mid s,a_{i}\right)\right|\leq 1$ for all $s,s^{\prime}\in\mathcal{S},a_{i}\in\mathcal{A}_{i}$.
We insert the result of ([132](#A3.E132)) back to ([131](#A3.E131)), and rewrite the result in vector form, we can achieve that

$$ $\displaystyle\beta_{i,h}\leq 3c_{\mathsf{b}}\sqrt{\frac{\log^{3}(\frac{KS\sum_{i=1}^{n}A_{i}}{\delta})}{KH}}\left(H\cdot 1+\mathsf{Var}_{\underline{\widehat{P}}_{i,h}^{\pi^{k},\widehat{V}}}\widehat{V}_{i,h+1}\right)$ $$

#### C.4.4 Proof of Lemma 7

To prove Lemma [7](#Thmlemma7), we start by analyzing the value function of policy $\pi$ under uncertainty set $\sigma_{i}$.
We first establish bounds on $\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$:

$$ $\displaystyle\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle=\min_{s\in\mathcal{S}}\sum_{k=1}^{K}\alpha_{k}^{K}\left[\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]\right]\right]$ $\displaystyle\geq\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right].$ $$

This follows from the robust Bellman equation ([13](#S3.E13)).

Next, we examine $\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$:

$$ $\displaystyle\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle=\max_{s\in\mathcal{S}}\sum_{k=1}^{K}\alpha_{k}^{K}\left[\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[r_{i,h}^{k}(s,a_{i})\right]+\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}(s)}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]\right]\right]$ $\displaystyle\leq 1+\sum_{k=1}^{K}\alpha_{k}^{K}\max_{(s,a_{i})\in\mathcal{S}\times\mathcal{A}_{i}}\left[\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{h,s,a_{i}}^{\pi_{-i}}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]\right].$ (133) $$

We now construct an auxiliary distribution vector $P^{\prime}_{h,s,a_{i}}\in\mathbb{R}^{S}$ by strictly reducing some elements of $P_{h,s,a_{i}}^{\pi_{-i}}$ such that:

$$ $0\leq P^{\prime}_{h,s,a_{i}}\leq P_{i,h,s,a_{i}}^{k}\quad\text{and}\quad\sum_{s^{\prime}\in\mathcal{S}}P_{i,h,s,a_{i}}^{k}(s^{\prime})-P^{\prime}_{h,s,a_{i}}(s^{\prime})=\left\|P^{\prime}_{h,s,a_{i}}-P_{i,h,s,a_{i}}^{k}\right\|_{1}=\sigma_{i}.$ (134) $$

Let $e_{s_{i,h}^{\star}}$ denote the standard basis vector supported on $s_{i,h}^{\star}$. We can show:

$$ $\displaystyle\frac{1}{2}\left\|P^{\prime}_{h,s,a_{i}}+\sigma_{i}\left[e_{s_{i,h}^{\star}}\right]^{\top}-P_{i,h,s,a_{i}}^{k}\right\|_{1}$ $\displaystyle\leq\frac{1}{2}\left\|P^{\prime}_{h,s,a_{i}}-P_{i,h,s,a_{i}}^{k}\right\|_{1}+\frac{1}{2}\left\|\sigma_{i}\left[e_{s_{i,h}^{\star}}\right]^{\top}\right\|_{1}\leq\sigma_{i},$ (135) $$

where the first inequality follows from the triangle inequality of the total variation distance.

From ([135](#A3.E135)), we conclude that:

$$ $\displaystyle\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{i,h,s,a_{i}}^{k}\right)}\mathcal{P}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]$ (136) $\displaystyle\leq\left(P^{\prime}_{h,s,a_{i}}+\sigma_{i}\left[e_{s_{i,h}^{\star}}\right]^{\top}\right)\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]$ $\displaystyle\leq\left\|P^{\prime}_{h,s,a_{i}}\right\|_{1}\left\|\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right\|_{\infty}+\sigma_{i}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s_{i,h+1}^{\star})\right]$ $\displaystyle\leq\left(1-\sigma_{i}\right)\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]+\sigma_{i}\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right].$ $$

Substituting ([136](#A3.E136)) into ([133](#A3.E133)) yields:

$$ $\displaystyle\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq 1+\left(1-\sigma_{i}\right)\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]+\sigma_{i}\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right].$ (137) $$

Combining ([C.4.4](#A3.Ex29b)) and ([137](#A3.E137)) gives:

$$ $\displaystyle\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ $\displaystyle\leq 1+\left(1-\sigma_{i}\right)\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}(s)\right]\right)$ $\displaystyle\leq 1+(1-\sigma_{i})\left[1+(1-\sigma_{i})\left(\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+2}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+2}^{\pi}(s)\right]\right)\right]$ $\displaystyle\leq\cdots\leq\frac{1-(1-\sigma_{i})^{H-h}}{\sigma_{i}}\leq\frac{1}{\sigma_{i}}.$ (138) $$

Combining this with the basic fact that $\max_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]\leq H-h+1$, we complete the proof.

#### C.4.5 Proof for Lemma 8

We introduce the following notation for the value function at time $h$:

$$ $\displaystyle\forall h\in[H],\quad\overline{V}_{i,h}^{\mathsf{span}}:=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]-\min_{s^{\prime}\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s^{\prime})\right],$ (139) $$

which normalizes the value function $\overline{V}_{i,h}^{\pi}$. This definition leads to the following bound:

$$ $\displaystyle\left\|\overline{V}_{i,h}^{\mathsf{span}}\right\|_{\infty}\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\},$ (140) $$

a result derived using Lemma [7](#Thmlemma7).
With this notation established, we now consider any transition kernel $P^{\prime}\in\mathbb{R}^{S}$ and any $\widetilde{P}\in\mathbb{R}^{S}$ such that $\widetilde{P}\in\mathcal{U}^{\sigma_{i}}(P^{\prime})$. For all $(i,h)\in[n]\times[H]$, we analyze the variance difference between the value functions under these kernels:

$$ $\displaystyle\left|\mathsf{Var}_{P^{\prime}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]\right)-\mathsf{Var}_{\widetilde{P}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]\right)\right|$ $\displaystyle=\big{|}\mathsf{Var}_{P^{\prime}}\left(\overline{V}_{i,h}^{\mathsf{span}}\right)-\mathsf{Var}_{\widetilde{P}}\left(\overline{V}_{i,h}^{\mathsf{span}}\right)\big{|}$ $\displaystyle\leq\big{\|}\widetilde{P}-P^{\prime}\big{\|}_{1}\left\|\overline{V}_{i,h}^{\mathsf{span}}\right\|_{\infty}$ $\displaystyle\leq\sigma_{i}\left(\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}\right)^{2}\leq\min\left\{\frac{1}{\sigma_{i}},H-h+1\right\}.$ (141) $$

#### C.4.6 Proof of Lemma 9

Analogous to Appendix [C.4.9](#A3.SS4.SSS9), we introduce some auxiliary values and reward functions to control

$$ $\sum_{j=h}^{H}\left<b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,j}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $$

as below for any time step $h$ and agent $i$.

###### Definition 3 .

For any time step $h\in[H]$ and the $i$-th agent, we denote $\overline{V}_{h}^{\min}:=\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ as the minimum value of all the entries in vector $\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]$. We further define $\overline{V}_{h}^{\prime}:=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]-\overline{V}_{h}^{\min}1$ as the truncated value function.
Eventually for reward function, we define $\overline{r}_{i,h}^{\min}=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\left(\overline{V}_{h+1}^{\min}-\overline{V}_{h}^{\min}\right)1$ as the truncated reward function..

Then applying the robust Bellman’s consistency equation in ([13](#S3.E13)) gives

$$ $\displaystyle\overline{V}_{h}^{\prime}=\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h}^{\pi}\right]-\overline{V}_{h}^{\min}1$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]-\overline{V}_{h}^{\min}1$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\left(\overline{V}_{h+1}^{\min}-\overline{V}_{h}^{\min}\right)1+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\overline{V}_{h+1}^{\prime}$ (142) $\displaystyle=\overline{r}_{i,h}^{\min}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\overline{V}_{h+1}^{\prime}.$ (143) $$

The above fact leads to

$$ $\displaystyle\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle\overset{\mathrm{(i)}}{=}\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}}\left(\overline{V}_{h+1}^{\prime}\right)$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\overline{V}_{h+1}^{\prime}\big{)}\circ\big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\overline{V}_{h+1}^{\prime}\big{)}$ $$

where (i) follows from the fact that $\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},V}}(V-b1)=\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},V}}(V)$ for any value vector $V\in\mathbb{R}^{S}$ and scalar $b$.
According to ([143](#A3.E143)) and ([13](#S3.E13)), we have

$$ $\displaystyle\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\Big{(}\overline{V}_{h}^{\prime}-\overline{r}_{i,h}^{\min}\Big{)}^{\circ 2}$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\overline{V}_{h}^{\prime}\circ\overline{V}_{h}^{\prime}+2\overline{V}_{h}^{\prime}\circ\overline{r}_{i,h}^{\min}-\overline{r}_{i,h}^{\min}\circ\overline{r}_{i,h}^{\min}$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\overline{V}_{h}^{\prime}\circ\overline{V}_{h}^{\prime}+2\|\overline{V}_{h}^{\prime}\|_{\infty}1,$ $$

where the last inequality arises from $\overline{r}_{i,h}^{\min}\leq\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi^{k}}r_{i,h}^{k}(\cdot,a_{i})\leq 1$ since $\overline{V}_{h+1}^{\min}-\overline{V}_{h}^{\min}\leq 0$ by definition.

Consequently, combining ([155](#A3.E155)) and the definition of $b_{h}^{j}$ in ([99](#A3.E99)), we arrive at

$$ $\displaystyle\sum_{j=h}^{H}\Big{<}b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},V}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\Big{>}$ $\displaystyle=\sum_{j=h}^{H}\left(b_{h}^{j}\right)^{\top}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\overline{V}_{j+1}^{\prime}\circ\overline{V}_{j+1}^{\prime}\right)-\overline{V}_{j}^{\prime}\circ\overline{V}_{j}^{\prime}+2\|\overline{V}_{h}^{\prime}\|_{\infty}1\right)$ $\displaystyle\overset{\mathrm{(i)}}{\leq}\sum_{j=h}^{H}\left[\left(b_{h}^{j}\right)^{\top}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},\overline{V}}\left(\overline{V}_{j+1}^{\prime}\circ\overline{V}_{j+1}^{\prime}\right)-\overline{V}_{j}^{\prime}\circ\overline{V}_{j}^{\prime}\right)\right]+2H\left\|\overline{V}_{h}^{\prime}\right\|_{\infty}$ $$

where (i) and the last inequality hold by the fact $\left\|\overline{V}_{h}^{\prime}\right\|_{\infty}\geq\left\|\overline{V}_{h+1}^{\prime}\right\|_{\infty}\geq\cdots\geq\left\|\overline{V}_{H}^{\prime}\right\|_{\infty}$. Further according to basic calculus, we have

$$ $\displaystyle\sum_{j=h}^{H}\Big{<}b_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\pi^{k},V}}\left(\mathbb{E}_{\pi\sim\widehat{\xi}}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\Big{>}$ $\displaystyle=\sum_{j=h}^{H}\left[\left(b_{h}^{j+1}\right)^{\top}\left(\overline{V}_{j+1}^{\prime}\circ\overline{V}_{j+1}^{\prime}\right)-(b_{h}^{j})^{\top}\left(\overline{V}_{j}^{\prime}\circ\overline{V}_{j}^{\prime}\right)\right]+2H\left\|\overline{V}_{h}^{\prime}\right\|_{\infty}$ $\displaystyle\leq\left\|b_{h}^{H+1}\right\|_{1}\left\|\overline{V}_{H+1}^{\prime}\circ\overline{V}_{H+1}^{\prime}\right\|_{\infty}+2H\left\|\overline{V}_{h}^{\prime}\right\|_{\infty}$ $\displaystyle\leq 3H\left\|\overline{V}_{h}^{\prime}\right\|_{\infty}.$ (144) $$

###### Definition 3 .

#### C.4.7 Proof of Lemma 10

To prove the inequality involving $P_{i,h}^{\widehat{\pi}_{-i}^{k},V}V$ and $\widehat{P}_{i,h}^{\widehat{\pi}_{-i}^{k},V}V$, we start by analyzing the absolute difference between these terms:

$$ $\displaystyle\left|P_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k},V}V-\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k},V}V\right|$ $\displaystyle=\left|\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}\right)}\mathcal{P}V-\inf_{\mathcal{P}\in\mathcal{U}^{\sigma_{i}}\left(\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}\right)}\mathcal{P}V\right|$ $\displaystyle\overset{(\mathrm{i})}{=}\left|\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}\max_{\alpha\in[\min_{s}V(s),\max_{s}V(s)]}\left[P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}-\sigma_{i}\left(\alpha-\min_{s^{\prime}}[V]_{\alpha}(s^{\prime})\right)\right]\right.$ $\displaystyle\quad\quad\quad-\left.\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}\max_{\alpha\in[\min_{s}V(s),\max_{s}V(s)]}\left[\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}-\sigma_{i}\left(\alpha-\min_{s^{\prime}}[V]_{\alpha}(s^{\prime})\right)\right]\right|$ $\displaystyle\leq\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}\max_{\alpha\in[\min_{s}V(s),\max_{s}V(s)]}\left|P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}-\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}\right|,$ (145) $$

where $(\mathrm{i})$ follows from applying the robust Bellman equation ([13](#S3.E13)), and the last inequality uses the fact that the maximum operator is 1-Lipschitz.

Next, we apply Bernstein’s inequality to bound the difference between $P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}$ and $\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}$ for fixed $\alpha$, $k$, and $(s,a_{i})$. With probability at least $1-\delta$, we have:

$$ $\displaystyle\left|P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}-\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}\right|$ $\displaystyle\leq\sqrt{\frac{2\log\left(\frac{2}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}}\left([V]_{\alpha}\right)}+\frac{2H\log\left(\frac{2}{\delta}\right)}{3N}.$ (146) $$

To extend this bound to all $(s,a_{i})$, we use a uniform bound over an $\varepsilon_{1}$-net for $\alpha$. The net size $|N_{\varepsilon_{1}}|\leq\frac{3H}{\varepsilon_{1}}$ allows us to apply the union bound:

$$ $\displaystyle\left|P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}-\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}\right|$ $\displaystyle\leq\max_{\alpha\in N_{\varepsilon_{1}}}\left|P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}-\widehat{P}_{i,h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}[V]_{\alpha}\right|+\varepsilon_{1}$ $\displaystyle\leq\sqrt{\frac{2\log\left(\frac{2S\sum_{i=1}^{n}A_{i}|N_{\varepsilon_{1}}|Kn}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}}(V)}+\frac{2H\log\left(\frac{2S\sum_{i=1}^{n}A_{i}|N_{\varepsilon_{1}}|Kn}{\delta}\right)}{3N}+\varepsilon_{1}$ $\displaystyle\leq\sqrt{\frac{2\log\left(\frac{2S\sum_{i=1}^{n}A_{i}NKn}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{P_{h,s,a_{i}}^{\widehat{\pi}_{-i}^{k}}}(V)}+\frac{H\log\left(\frac{2S\sum_{i=1}^{n}A_{i}NKn}{\delta}\right)}{N},$ (147) $$

where the last steps use that $\varepsilon_{1}=\frac{H\log\left(\frac{2S\sum_{i=1}^{n}A_{i}NKn}{\delta}\right)}{3N}$ and $|N_{\varepsilon_{1}}|\leq 9N$.

Inserting this back into ([145](#A3.E145)) gives:

$$ $\displaystyle\left|P_{i,h}^{\widehat{\pi}_{-i}^{k},V}V-\widehat{P}_{i,h}^{\widehat{\pi}_{-i}^{k},V}V\right|$ $\displaystyle\leq\sqrt{\frac{2\log\left(\frac{2S\sum_{i=1}^{n}A_{i}NKn}{\delta}\right)}{N}}\sqrt{\mathsf{Var}_{P_{h}^{\widehat{\pi}_{-i}^{k}}}(V)}+\frac{H\log\left(\frac{2S\sum_{i=1}^{n}A_{i}NKn}{\delta}\right)}{N}1$ $\displaystyle\leq 3\sqrt{\frac{H^{2}\log\left(\frac{2S\sum_{i=1}^{n}A_{i}NKn}{\delta}\right)}{N}}1.$ $$

This completes the proof by showing that the bound holds uniformly over all $(s,a_{i})\in\mathcal{S}\times\mathcal{A}_{i}$.

#### C.4.8 Proof of Lemma 11

Before proving Lemma [11](#Thmlemma11), we first state a modified version of the Freedman inequality for martingales, which is crucial for our analysis.

###### Theorem 5 .

Suppose $Y_{n}=\sum_{k=1}^{n}X_{k}\in\mathbb{R}$, where $\{X_{k}\}$ is a real-valued scalar sequence such that

$$ $\left|X_{k}\right|\leq R\qquad\text{and}\qquad\mathbb{E}\left[X_{k}\mid\{X_{j}\}_{j<k}\right]=0\quad\text{for all }k\geq 1$ $$

for some constant $R>0$. Define

$$ $W_{n}:=\sum_{k=1}^{n}\mathbb{E}_{k-1}[X_{k}^{2}],$ $$

where $\mathbb{E}_{k-1}$ denotes the conditional expectation given $\{X_{j}\}_{j<k}$. For any $\kappa>0$, with probability at least $1-\delta$, the following holds:

$$ $\displaystyle\left|Y_{n}\right|$ $\displaystyle\leq\sqrt{8W_{n}\log\frac{3n}{\delta}}+5R\log\frac{3n}{\delta}\leq\kappa W_{n}+\left(\frac{2}{\kappa}+5R\right)\log\frac{3n}{\delta}.$ (148) $$

###### Proof.

Suppose deterministically that $W_{n}\leq\sigma^{2}$ for some $\sigma^{2}$. According to Li et al., (), with probability at least $1-\delta$, we have

$$ $\displaystyle\left|Y_{n}\right|$ $\displaystyle\leq\sqrt{8\max\left\{W_{n},\frac{\sigma^{2}}{2^{K}}\right\}\log\frac{2K}{\delta}}+\frac{4}{3}R\log\frac{2K}{\delta}.$ $$

for any positive integer $K\geq 1$. Utilizing the trivial bound $W_{n}\leq nR^{2}$, set $\sigma^{2}=nR^{2}$ and $K=\log_{2}n$. Then:

$$ $\displaystyle\left|Y_{n}\right|$ $\displaystyle\leq\sqrt{8\max\left\{W_{n},R^{2}\right\}\log\frac{4\log_{2}n}{\delta}}+\frac{4}{3}R\log\frac{4\log_{2}n}{\delta}$ $\displaystyle\leq\sqrt{8W_{n}\log\frac{3n}{\delta}}+\sqrt{8R^{2}\log\frac{3n}{\delta}}+\frac{4}{3}R\log\frac{3n}{\delta}$ $\displaystyle\leq\sqrt{8W_{n}\log\frac{3n}{\delta}}+5R\log\frac{3n}{\delta},$ $$

where we used $4\log_{2}n\leq 3n$ for any integer $n\geq 1$. This establishes the first inequality in ([148](#A3.E148)). The second inequality follows from the elementary inequality $2ab\leq a^{2}+b^{2}$.
∎

To prove Lemma [11](#Thmlemma11), we apply Lemma [5](#Thmtheorem5). Define

$$ $\displaystyle R$ $\displaystyle:=\max_{k\in[K]}\left|\alpha_{k}^{K}\left<\widehat{\pi}_{i,h}^{k}(s),r_{i,h}^{k}(s,\cdot)\right>\right|\leq\left\{\max_{k\in[K]}\alpha_{k}^{K}\right\}\left\{\max_{k\in[K]}\|\widehat{\pi}_{i,h}^{k}(s)\|_{1}\|r_{i,h}^{k}\|_{\infty}\right\}\leq\frac{2c_{\alpha}\log K}{K},$ $$

where the first line uses Lemma [19](#Thmlemma19). We further define

$$ $\displaystyle W_{K}$ $\displaystyle=\sum_{k=1}^{K}(\alpha_{k}^{K})^{2}\mathsf{Var}_{h,k-1}\left(\left<\widehat{\pi}_{i,h}^{k}(s),r_{i,h}^{k}(s,\cdot)\right>\right)$ $\displaystyle\leq\left\{\max_{k\in[K]}\alpha_{k}^{K}\right\}\left\{\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{h,k-1}\left(\left<\widehat{\pi}_{i,h}^{k}(s),r_{i,h}^{k}(s,\cdot)\right>\right)\right\}$ $\displaystyle\leq\frac{2c_{\alpha}\log K}{K}\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{h,k-1}(r_{i,h}^{k}(s)),$ $$

where we use variance operator $\mathsf{Var}_{h,k-1}[\cdot]$ to denote the variance conditional on what happens before the beginning of the $k$-th round of data collection for step $h$.
Applying Freedman’s inequality (Lemma [5](#Thmtheorem5)) with $\kappa_{1}=\sqrt{K\log(K/\delta)}$, we obtain

$$ $\displaystyle\left|\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{k}(s,a_{i})-\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{\widehat{\pi}_{-i}^{k}}(s,a_{i})\right|$ $\displaystyle\leq\kappa_{1}W_{K}+\left(\frac{2}{\kappa_{1}}+5R_{1}\right)\log\left(\frac{3K}{\delta}\right)$ $\displaystyle\leq 2c_{\alpha}\sqrt{\frac{\log^{3}\left(\frac{K}{\delta}\right)}{K}}\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{h,k-1}\left(r_{i,h}^{k}(s)\right)+\left(2\sqrt{\frac{1}{K\log\left(\frac{K}{\delta}\right)}}+\frac{10c_{\alpha}\log K}{K}\right)\log\left(\frac{3K}{\delta}\right)$ $\displaystyle\leq 2c_{\alpha}\sqrt{\frac{\log^{3}\left(\frac{K}{\delta}\right)}{K}}\sum_{k=1}^{K}\alpha_{k}^{K}\mathsf{Var}_{h,k-1}\left(r_{i,h}^{k}(s)\right)+4\sqrt{\frac{\log\left(\frac{3K}{\delta}\right)}{K}},$ $$

with probability at least $1-\delta$. Taking a union bound over all $s\in\mathcal{S}$, there exists an absolute constant $c_{r}$ such that

$$ $\displaystyle\sum_{k=1}^{K}\alpha_{k}^{K}\left|r_{i,h}^{\widehat{\pi}^{k}}-\overline{r}_{i,h}^{\widehat{\pi}^{k}}\right|\leq c_{r}\sqrt{\frac{\log(KS/\delta)}{K}}1.$ $$

###### Theorem 5 .

###### Proof.

#### C.4.9 Proof of Lemma 13

In this section, we want to take the accessible range of the robust value function $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]$ into consideration when controlling $\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$. Towards this, we introduce some auxiliary values and reward functions as below.

###### Definition 4 .

For any time step $h\in[H]$ and the $i$-th agent, we denote $\overline{V}_{h}^{\min}:=\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}(s)\right]$ as the minimum value of all the entries in vector $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]$. We further define $\overline{V}_{h}^{\prime}:=\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]-\overline{V}_{h}^{\min}1$ as the truncated value function.
Eventually for reward function, we define $\overline{r}_{i,h}^{\min}=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\left(\overline{V}_{h+1}^{\min}-\overline{V}_{h}^{\min}\right)1$ as the truncated reward function..

With above notation, we introduce the following fact of $\overline{V}_{h}^{\prime}$:

$$ $\displaystyle\overline{V}_{h}^{\prime}$ $\displaystyle=\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]-\overline{V}_{h}^{\min}1$ $\displaystyle\overset{\mathrm{(i)}}{=}\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\overline{V}_{h}^{\min}1$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]$ $\displaystyle\quad+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]-\overline{V}_{h}^{\min}1$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})+\left(\overline{V}_{h+1}^{\min}-\overline{V}_{h}^{\min}\right)1$ $\displaystyle\quad+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\overline{V}_{h+1}^{\prime}+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]$ $\displaystyle=\overline{r}_{i,h}^{\min}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\overline{V}_{h+1}^{\prime}+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right],$ (149) $$

where (i) holds by the robust Bellman’s consistency equation of $\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h}^{\pi}\right]$. With the above fact in hand, we control $\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ as follows:

$$ $\displaystyle\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle\overset{\mathrm{(i)}}{=}\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}\left(\overline{V}_{h+1}^{\prime}\right)$ (150) $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\overline{V}_{h+1}^{\prime}\big{)}\circ\big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\overline{V}_{h+1}^{\prime}\big{)},$ $$

where (i) follows from the fact that $\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}(V-b1)=\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}(V)$ for any value vector $V\in\mathbb{R}^{S}$ and scalar $b$,
Additionally according to ([149](#A3.E149)), we have

$$ $\displaystyle\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\Big{(}\overline{V}_{h}^{\prime}-\overline{r}_{i,h}^{\min}-\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\Big{)}^{\circ 2}$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\overline{V}_{h}^{\prime}\circ\overline{V}_{h}^{\prime}+2\overline{V}_{h}^{\prime}\circ\Big{(}\overline{r}_{i,h}^{\min}+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\Big{)}$ $\displaystyle-\Big{(}\overline{r}_{i,h}^{\min}+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\Big{)}^{\circ 2}$ $$

Furthermore, we have

$$ $\displaystyle\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\right)$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\overline{V}_{h}^{\prime}\circ\overline{V}_{h}^{\prime}+2\overline{V}_{h}^{\prime}\circ\Big{(}\overline{r}_{i,h}^{\min}+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\Big{)}$ $\displaystyle\quad-\Big{(}\overline{r}_{i,h}^{\min}+\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\Big{)}^{\circ 2}$ $\displaystyle\overset{\mathrm{(i)}}{\leq}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\overline{V}_{h}^{\prime}\circ\overline{V}_{h}^{\prime}+2\big{\|}\overline{V}_{h}^{\prime}\big{\|}_{\infty}\Big{(}1+\Big{|}\Big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{\widehat{P}}_{i,h}^{\widehat{\pi}^{k},\overline{V}}-\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\Big{)}\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,h+1}^{\pi}\right]\Big{|}\Big{)}$ (151) $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{h+1}^{\prime}\circ\overline{V}_{h+1}^{\prime}\right)-\overline{V}_{h}^{\prime}\circ\overline{V}_{h}^{\prime}+2\big{\|}\overline{V}_{h}^{\prime}\big{\|}_{\infty}1+6\|V_{h}^{\prime}\|_{\infty}\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}1,$ (152) $$

holds with probability at least $1-\delta$, where (i) arises from $\underline{r}_{i,h}^{\min}\leq\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\widehat{\pi}_{i,h}^{k}}r_{i,h}^{k}(\cdot,a_{i})\leq 1$ since $V_{h+1}^{\min}-V_{h}^{\min}\leq 0$ by definition, and the last inequality holds by Lemma [10](#Thmlemma10).
Finally, combining ([152](#A3.E152)) and the definition of $d_{h}^{j}$ in ([99](#A3.E99)), the term of interest can be controlled as

$$ $\displaystyle\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle=\sum_{j=h}^{H}(d_{h}^{j})^{\top}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{j+1}^{\prime}\circ\overline{V}_{j+1}^{\prime}\right)-\overline{V}_{j}^{\prime}\circ\overline{V}_{j}^{\prime}+2\|\overline{V}_{j}^{\prime}\|_{\infty}1+6\|\overline{V}_{j}^{\prime}\|_{\infty}\sqrt{\frac{H^{2}\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}1\right)$ $\displaystyle\overset{\mathrm{(i)}}{\leq}\sum_{j=h}^{H}\left[(d_{h}^{j})^{\top}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}\left(\overline{V}_{j+1}^{\prime}\circ\overline{V}_{j+1}^{\prime}\right)-\overline{V}_{j}^{\prime}\circ\overline{V}_{j}^{\prime}\right)\right]+2H\|\overline{V}_{h}^{\prime}\|_{\infty}$ $\displaystyle\quad+6H^{2}\|\overline{V}_{h}^{\prime}\|_{\infty}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}$ $$

where (i) holds by the fact $\|\overline{V}_{h}^{\prime}\|_{\infty}\geq\|\overline{V}_{h+1}^{\prime}\|_{\infty}\geq\cdots\geq\|\overline{V}_{H}^{\prime}\|_{\infty}$. With further basic calculus, we can finally obtain that

$$ $\displaystyle\sum_{j=h}^{H}\left<d_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},\overline{V}}}\left(\mathbb{E}_{\pi\sim\zeta}\left[\overline{V}_{i,j+1}^{\pi}\right]\right)\right>$ $\displaystyle=\sum_{j=h}^{H}\left[(d_{h}^{j+1})^{\top}\left(\overline{V}_{j+1}^{\prime}\circ\overline{V}_{j+1}^{\prime}\right)-(d_{h}^{j})^{\top}\left(\overline{V}_{j}^{\prime}\circ\overline{V}_{j}^{\prime}\right)\right]+2H\|\overline{V}_{h}^{\prime}\|_{\infty}+6H^{2}\|\overline{V}_{h}^{\prime}\|_{\infty}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}$ $\displaystyle\leq\left\|d_{h}^{H+1}\right\|_{1}\left\|\overline{V}_{H+1}^{\prime}\circ\overline{V}_{H+1}^{\prime}\right\|_{\infty}+2H\|\overline{V}_{h}^{\prime}\|_{\infty}+6H^{2}\|\overline{V}_{h}^{\prime}\|_{\infty}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}$ $\displaystyle\leq 3H\|\overline{V}_{h}^{\prime}\|_{\infty}+6H^{2}\|\overline{V}_{h}^{\prime}\|_{\infty}\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}$ $\displaystyle=3H\|\overline{V}_{h}^{\prime}\|_{\infty}\left(1+2H\sqrt{\frac{\log\left(\frac{18S\sum_{i=1}^{n}A_{i}nHNK}{\delta}\right)}{N}}\right),$ $$

###### Definition 4 .

#### C.4.10 Proof of Lemma 16

Analogous to Appendix [C.4.9](#A3.SS4.SSS9), we introduce some auxiliary values and reward functions to control

$$ $\sum_{j=h}^{H}\left<w_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi}\right]\right)\right>$ $$

as below: for any time step $h$ and the $i$-th agent

###### Definition 5 .

For any time step $h\in[H]$ and the $i$-th agent, we denote $V_{h}^{\min}:=\min_{s\in\mathcal{S}}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}(s)\right]$ as the minimum value of all the entries in vector $\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi}\right]$. We further define $V_{h}^{\prime}:=C_{i,h}^{\pi}-V_{h}^{\min}1$ as the truncated value function.
Eventually for reward function, we define $r_{i,h}^{\min}=\sum_{k=1}^{K}\alpha_{k}^{K}\mathbb{E}_{a_{i}\sim\pi_{i,h}^{k}}r_{i,h}^{\widehat{\pi}^{k}_{-i}}(\cdot,a_{i})+\left(V_{h+1}^{\min}-V_{h}^{\min}\right)1$ as the truncated reward function..

Then applying the robust Bellman’s consistency equation in ([13](#S3.E13)) gives

$$ $\displaystyle V_{h}^{\prime}=\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h}^{\pi,\sigma_{i}}\right]-V_{h}^{\min}1$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi,\sigma_{i}}\right]-V_{h}^{\min}1$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}r_{i,h}^{\widehat{\pi}^{k}}+\left(V_{h+1}^{\min}-V_{h}^{\min}\right)1+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}V_{h+1}^{\prime}$ (153) $\displaystyle=r_{i,h}^{\min}+\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}V_{h+1}^{\prime}.$ (154) $$

The above fact leads to

$$ $\displaystyle\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,h+1}^{\pi,\sigma_{i}}\right]\right)$ $\displaystyle\overset{\mathrm{(i)}}{=}\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}}(V_{h+1}^{\prime})$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\left(V_{h+1}^{\prime}\circ V_{h+1}^{\prime}\right)-\big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}V_{h+1}^{\prime}\big{)}\circ\big{(}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}V_{h+1}^{\prime}\big{)}$ $\displaystyle\overset{\mathrm{(ii)}}{=}\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\left(V_{h+1}^{\prime}\circ V_{h+1}^{\prime}\right)-\Big{(}V_{h}^{\prime}-r_{i,h}^{\min}\Big{)}^{\circ 2}$ $\displaystyle=\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\left(V_{h+1}^{\prime}\circ V_{h+1}^{\prime}\right)-V_{h}^{\prime}\circ V_{h}^{\prime}+2V_{h}^{\prime}\circ r_{i,h}^{\min}-r_{i,h}^{\min}\circ r_{i,h}^{\min}$ $\displaystyle\leq\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}\left(V_{h+1}^{\prime}\circ V_{h+1}^{\prime}\right)-V_{h}^{\prime}\circ V_{h}^{\prime}+2\|V_{h}^{\prime}\|_{\infty}1,$ (155) $$

where (i) follows from the fact that $\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}}(V-b1)=\mathrm{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}}(V)$ for any value vector $V\in\mathbb{R}^{S}$ and scalar $b$, (ii) holds by ([154](#A3.E154)) and ([13](#S3.E13)), and the last inequality arises from $r_{i,h}^{\min}\leq r_{i,h}^{\pi}\leq 1$ since $V_{h+1}^{\min}-V_{h}^{\min}\leq 0$ by definition.

Consequently, combining ([155](#A3.E155)) and the definition of $w_{h}^{j}$ in ([108](#A3.E108)), we arrive at

$$ $\displaystyle\sum_{j=h}^{H}\Big{<}w_{h}^{j},\mathsf{Var}_{\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,h}^{\widehat{\pi}^{k},V}}\left(\mathbb{E}_{\pi\sim\zeta}\left[V_{i,j+1}^{\pi,\sigma_{i}}\right]\right)\Big{>}$ $\displaystyle\leq\sum_{j=h}^{H}(w_{h}^{j})^{\top}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}\left(V_{j+1}^{\prime}\circ V_{j+1}^{\prime}\right)-V_{j}^{\prime}\circ V_{j}^{\prime}+2\|V_{h}^{\prime}\|_{\infty}1\right)$ $\displaystyle\overset{\mathrm{(i)}}{\leq}\sum_{j=h}^{H}\left[(w_{h}^{j})^{\top}\left(\sum_{k=1}^{K}\alpha_{k}^{K}\underline{P}_{i,j}^{\widehat{\pi}^{k},V}\left(V_{j+1}^{\prime}\circ V_{j+1}^{\prime}\right)-V_{j}^{\prime}\circ V_{j}^{\prime}\right)\right]+2H\|V_{h}^{\prime}\|_{\infty}$ $\displaystyle=\sum_{j=h}^{H}\left[(w_{h}^{j+1})^{\top}\left(V_{j+1}^{\prime}\circ V_{j+1}^{\prime}\right)-(w_{h}^{j})^{\top}\left(V_{j}^{\prime}\circ V_{j}^{\prime}\right)\right]+2H\|V_{h}^{\prime}\|_{\infty}$ $\displaystyle\leq\|w_{h}^{H+1}\|_{1}\left\|V_{H+1}^{\prime}\circ V_{H+1}^{\prime}\right\|_{\infty}+2H\|V_{h}^{\prime}\|_{\infty}$ $\displaystyle\leq 3H\|V_{h}^{\prime}\|_{\infty},$ (156) $$

where (i) and the last inequality hold by the fact $\|V_{h}^{\prime}\|_{\infty}\geq\|V_{h+1}^{\prime}\|_{\infty}\geq\cdots\geq\|V_{H}^{\prime}\|_{\infty}$ and basic calculus.

###### Definition 5 .

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What is the $(s, \mathbf{a})$-rectangularity condition? An explicit definition for this would be useful to readers with limited background in this class of problems.  Can the authors provide a brief formal definition of the $(s,a)$-rectangularity condition in the main text, along with a sentence or two explaining its significance in robust MDPs/Markov games?
2. How are the uncertainty sets defined in this work practically meaningful? Can the authors to provide 1-2 concrete examples of how their fictitious uncertainty sets could model real-world uncertainties in multi-agent systems, and how this compares to uncertainty modeling in previous work?
3. What are the real-world scenarios which are modelled well by this class of RMGs with fictitious uncertainty sets?
4. When the authors compare with prior art in Table 1, aren’t the game models also different in the different works? Are there any trade-offs due to this?
5. Is the sample complexity improvement obtained by the authors primarily dependent on their robust Markov game definition which is different from the other works in literature?
6. Doesn’t the uncertainty set become very large and potentially intractable in problems with large state spaces?
7. Is it necessary that the uncertainty set be defined as part of the problem? In practical use cases, how does one define these sets? Specifically, can the authors discuss guidelines or heuristics for how practitioners could define appropriate uncertainty sets for real-world multi-agent problems, and whether there are ways to learn or adapt the uncertainty sets from data?
8. While I agree that the proofs are too long to be included in the main paper, but considering the fact that the primary contribution of this work is theoretical, it would be very helpful if the authors can provide short proof sketches or key proof ideas for all their claims in the main paper itself.
soundness: 2
strengths: 1. The paper is well written. All concepts and claims are clearly defined and claims are backed by theoretical justifications.
2. While the authors provide a detailed theoretical analysis, I am unable to verify the correctness of their results, though I could not find any error.
3. The authors clearly state their algorithm which can be applied to practical problems.
summary: The authors propose a new multi-agent problem class, which is a new class of robust Markov Games with fictitious uncertainty sets. The authors define solution concepts – robust Nash equilibrium and robust coarse correlated equilibrium for this new class of games and also prove the existence of these. The authors then propose a novel algorithm called Robust-Q-FTRL to find robust CCE for their class of robust Markov Games and also establish its sample complexity. Using their approach in their class if problems, the authors break the curse of multi-agency in MARL.
weaknesses: 1. The authors do not provide any numerical examples. While they claim that their approach is scalable, some numerical examples will be useful to understand scalability and the sample efficiency claims better. Can the authors include a small simulation study to demonstrate the scalability and sample efficiency of their approach compared to baselines, even if only on a toy problem?
2. The authors do not provide any comparison of their approach with other approaches in literature in terms of numerical computations.
3. The authors do not comment on whether the policies learned using their algorithms are strategy proof from a learning perspective, i.e., do they perform well when other agents use different learning algorithms.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Questions:

1. Does the definition of the policy imply that the final solution can be an un-stationary policy?
2. Are the fictitious uncertainty sets un-stationary? That is, does each period have a distinct uncertainty set?
3. In a robust Markov game (MG) setting where the uncertainty set differs at each time step, is it valid to interchange the max operators in the definition of the optimal robust value function at each time step?
4. You assume that sampling from the true nominal transition kernel is possible. However, if this is feasible, why is robustness necessary? To address the sim-to-real gap, wouldn’t it be more appropriate to assume that sampling from the true nominal transition kernel isn’t possible?
5. Additionally, could you explain why the estimation method in Algorithm 1 differs from that in standard MG settings?
6. My understanding of the fictitious uncertainty set is that it involves a policy-based adjustment to the classic reference transition kernel (nominal transition kernel), which is then used as the basis for constructing the uncertainty set. This implies two rounds of adjustment in creating the uncertainty set, adding an extra step compared to traditional uncertainty sets. Does this make the new uncertainty set more conservative? How do you quantify the difference between the fictitious uncertainty set and a conventional uncertainty set without the initial adjustment?
7. Could you elaborate on the reason for breaking the curse of multiagency? Is it mainly due to the FTRL algorithm, or does the fictitious uncertainty set contribute to this outcome?
8. Could additional experimental results be provided to verify the convergence of the algorithm and demonstrate that the robust policy in this setting achieves better robustness compared to the classical (s, a)-rectangularity setting?
9. Line 65, Kearns & Singh, 1999 is not about solving finite-horizon multi-player general-sum Markov games, why cite this?
soundness: 3
strengths: Strengths:

1. The main body of this paper is well-written.
2. The proposed uncertainty set is interesting.
3. The proposed algorithm effectively reduces sample complexity, avoiding the exponential growth issue in multi-agent systems caused by the joint action space.
summary: This paper addresses the challenge of the “curse of multiagency” in robust MARL by introducing a new class of robust Markov games with a fictitious uncertainty set. This kind of uncertainty set allows any one agent to consider uncertainties arising from both the environment and the policies of other agents. Then, the authors prove the existence of robust Nash equilibrium and propose an algorithm Robust-Q-FTRL, which learns an approximate coarse correlated equilibrium with polynomial sample complexity, thus breaking the curse of multiagency in RMG for the first time. No experiment validation is provided.
weaknesses: Weakness:

1. The approach assumes access to a generative model with a true nominal transition kernel for sample generation, which is infeasible in real-world applications.
2. Although the paper provides theoretical evidence for overcoming the curse of multiagency, further experimental validation is needed to determine whether the algorithm itself breaks the curse or if the new problem formulation inherently possesses this advantage, potentially enabling other algorithms to achieve similar results. In addition, additional experimental evidence is needed to demonstrate that the new uncertainty set yields more robust policies compared to the previously classic (s, a)-rectangularity uncertainty sets.
3. While reading through the proof section, I noticed typos and unclear expressions, e.g., on lines 1044, 1170, and 1175. Additionally, the proofs lack tight cohesion between sections as seen in the main body, and there are missing references to some essential equations, which makes the reading experience somewhat challenging.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could the authors discuss the challenges involved in deriving sample complexity bounds for robust NE within the existing framework? Additionally, proposing the derivation of robust NE sample complexity as a direction for future work would help address the question of completeness and open avenues for further research.
2. Could the authors include a concrete example or comparison showing how the policy-induced $(s, a_i)$-rectangularity condition improves sample complexity relative to the $(s, a)$-rectangularity condition used in previous work? Such a comparison would help readers understand the practical impact of this new approach.
3. Could the authors clarify why separate constraints on $N$ and $K$ are presented, rather than combining them as $N = KH$? If there are specific advantages or insights gained from keeping these constraints separate, an explanation would enhance the clarity and completeness of the analysis.
4. The paper lacks clear organization and contains numerous typos. For instance, on page 3, it refers to "Section 2.2 for details," whereas the relevant content is actually in Section 3.1. Additionally, the algorithm description does not appear until page 8. A thorough proofreading pass to correct section references and typographical errors is also recommended to enhance readability.
soundness: 3
strengths: This paper proposes a sample-efficient algorithm for robust general-sum and Markov games (RMGs), breaking the curse of dimensionality in RMGs and achieving optimal sample complexity for CCE with respect to the action space size.
summary: This paper proposes a sample-efficient algorithm for robust general-sum and Markov games (RMGs), breaking the curse of dimensionality in RMGs and achieving optimal sample complexity for CCE with respect to the action space size.
weaknesses: The sample complexity optimality with respect to $A_i$ is achieved at the expense of an increased horizon $H$. Additionally, while the existence of a robust Nash equilibrium (NE) is proven, only the sample complexity for robust CCE is derived, which affects the completeness of the work.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Is the proposed Algorithm 1 overall computationally tractable? I found a related discussion in Sec. 4.2, but it is about the Q value estimation.
soundness: 2
strengths: The robust multi-agent learning setting is interesting and reasonable to me. 

Although under a different setting compared with previous works considering rectangular uncertainty set, the sample complexity results "break the curse of multi-agency".
summary: This paper studies robust reinforcement learning in the multi-agent setting. The authors propose a new uncertainty set called "fictitious uncertainty set", where depends on the joint policy of agents. Based on the new notion, they establish the existence of robust NE and CCE. After that, they propose the Robust-Q-FTRL algorithm, and under the generative model assumption, they establish sample complexity only scaling with the number of actions of all agents.
weaknesses: 1. **About paper writing**: I think there are several parts of the paper writing should be improved to avoid confusion.

* If I understand correctly, starting from Section 3.2, the definition of value functions are associated with the fictitious RMGs, instead of the rectangular uncertainty sets. In another word, one should interpret $V^{\pi,\sigma_i}_{i,h}$ through the definition in Eq.(26), instead of Eq. (5), which are different in the uncertainty set w.r.t. the inf operator.

    However, I did not find a declaration of this abuse of notation (and personally, I would recommend use different notations to avoid confusion). This is crucial because it makes significant difference in how to interpret the results. Under the definition of Eq.(26), the uncertainty set varies for different policies $\pi$, rather than that one first fix the uncertainty set by an arbitrary reference policy $\pi$ and then use that to define the NE/CCE.

* The "technical insights" paragraph in Section 4.3 does not explain clearly how the proposed algorithms avoid the curse of multi-agency. This is crucial to evaluate the technical novelty of this work (also see my second point in the following).

2. **The fundamental reason for avoiding curse of multi-agency is not clear to me**. 

* Comparing with previous works, this paper consider a different uncertainty set, it is not clear to me whether it is the key variation making the problem more tractable than before. Especially, by definition, the new uncertainty set only quantifies the uncertainty of the (weighted) marginal transition function. If this is the essential reason, claiming avoiding curse of multi-agency as advantage over previous work would be unfair.

* Besides, this paper relies on generative model assumption, I'm also curious whether it is a common assumption in previous literature?

3. **The proposed uncertainty set may not be sufficient to quantify the uncertainty**. I'm curious what would be the benefits of considering new uncertainty set, and is it reasonable to consider such set in practice? For example, under what scenarios (what kind of model difference between simulator and practice), the propose uncertainty set would be useful.

Besides, technically speaking, the new uncertainty set may be insufficient to capture the sim-to-real gap. Note that the ratio $\pi_h(a_i,a_{-i}|s) / \pi_{i,h}(a_i|s)$ could varies a lot for different $a_{-i}$, given that this paper considers correlated policies. Consider the scenario where the sim-to-real gaps of $P^0_{h,s,\textbf{a}}$ are very large for some $(s,\textbf{a})$, while, coincidently, the learned CCE only has low ratio $\pi_h(a_i,a_{-i}|s) / \pi_{i,h}(a_i|s)$ on them. In that case, fictitious uncertainty set is not sufficient to quantify the uncertainty.

Meta-review: This paper investigates the distributionally robust Markov games (RMGs) settings and focuses on a new uncertainty set called "fictitious uncertainty set". Then, the authors propose an algorithm named Robust-Q-FTRL to find robust approximate CCE and provide theoretical guarantees about the upper bound of sample complexity. The sample complexity shows that it breaks the curse of multi-agency in MARL.

The paper raises novel settings and provides detailed theoretical guarantees. However, as challenged by the reviewers, how the new uncertainty set notion "fictitious uncertainty set" can capture real-world applications is not well justified. I suggest elaborating further on the reasons for "breaking the curse of multi-agency in MARL" and explaining how the new uncertainty set notion relates to applications in the future.

Decision: Reject