SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Example 2.8 ( $\mathcal{S} \times \mathcal{A}$ -rectangular robust MDP with neural function approximations). We consider an infinite state space  $\mathcal{S} \times \mathcal{A}$ -rectangular robust MDP whose realizable model space  $\mathcal{P}_{\mathrm{M}}$  is parameterized by an overparameterized neural network. We first define a two-layer fully-connected neural network on some  $\mathcal{X} \subseteq \mathbb{R}^{d_{\mathcal{X}}}$  as
$$
\mathrm {N N} (\mathbf {x}; \mathbf {W}, \mathbf {a}) = \frac {1}{\sqrt {m}} \sum_ {j = 1} ^ {m} a _ {j} \sigma \left(\mathbf {x} ^ {\top} \mathbf {w} _ {j}\right), \quad \forall \mathbf {x} \in \mathcal {X}, \tag {2.8}
$$
where  $m \in \mathbb{N}_+$  is the number of hidden units of NN,  $(\mathbf{W}, \mathbf{a})$  is the parameters given by  $\mathbf{W} = (\mathbf{w}_1, \dots, \mathbf{w}_m) \in \mathbb{R}^{d \times m}$ ,  $\mathbf{a} = (a_1, \dots, a_m)^\top \in \mathbb{R}^m$ , and  $\sigma(\cdot)$  is the activation function. Now we assume that the state space  $\mathcal{S} \subseteq \mathbb{R}^{d_S}$  for some  $d_S \in \mathbb{N}_+$ . Also, we identify actions via one-hot vectors in  $\mathbb{R}^{|\mathcal{A}|}$ , i.e., we represent  $a \in \mathcal{A}$  by  $(0, \dots, 0, 1, 0, \dots, 0)$  with 1 in the  $a$ -th coordinate. Let  $\mathcal{X} = \mathcal{S} \times \mathcal{A} \times \mathcal{S}$  with  $d_{\mathcal{X}} = 2d_{\mathcal{S}} + |\mathcal{A}|$ . Then an  $\mathcal{S} \times \mathcal{A}$ -rectangular robust MDP with neural function approximation is defined as an  $\mathcal{S} \times \mathcal{A}$ -rectangular robust MDP with  $\mathcal{P}_{\mathrm{M}}$  given by
$$
\mathcal {P} _ {\mathrm {M}} = \left\{P \left(s ^ {\prime} \mid s, a\right) = \mathrm {N N} \left(\left(s, a, s ^ {\prime}\right); \mathbf {W}, \mathbf {a} ^ {0}\right): \| \mathbf {W} - \mathbf {W} ^ {0} \| _ {2} \leq B _ {\mathrm {N}} \right\}, \tag {2.9}
$$
for some  $B_{\mathrm{N}} > 0$  and some fixed  $(\mathbf{W}^0,\mathbf{a}^0)$  which can be interpreted as the initialization. We refer to Appendix D.4.1 for more details about neural function approximations and analysis techniques.
Example 2.9 ( $S \times \mathcal{A}$ -rectangular robust factored MDP). We consider a factored MDP equipped with  $S \times \mathcal{A}$ -rectangular factored robust set. A standard factored MDP [17] is defined as follows. Let  $d$  be an integer and  $\mathcal{O}$  be a finite set. The state space  $S$  is factored as  $S = \mathcal{O}^d$ . For each  $i \in [d]$ ,  $s[i]$  is the  $i$ -coordinate of  $s$  and it is only influenced by  $s[\mathrm{pa}_i]$ , where  $\mathrm{pa}_i \subseteq [d]$ . That is, the transition of a factored MDP can be factorized as
$$
P _ {h} ^ {\star} (s ^ {\prime} | s, a) = \prod_ {i = 1} ^ {d} P _ {h, i} ^ {\star} (s ^ {\prime} [ i ] | s [ p a _ {i} ], a).
$$
Here we let the realizable model space  $\mathcal{P}_{\mathrm{M}}$  consist of all the factored transition kernels, i.e.,
$$
\mathcal {P} _ {\mathrm {M}} = \left\{P (s ^ {\prime} | s, a) = \prod_ {i = 1} ^ {d} P _ {i} (s ^ {\prime} [ i ] | s [ \mathrm {p a} _ {i} ], a): P _ {i}: \mathcal {S} [ \mathrm {p a} _ {i} ] \times \mathcal {A} \mapsto \Delta (\mathcal {O}), \forall i \in [ d ] \right\}.
$$
For an  $\mathcal{S} \times \mathcal{A}$ -rectangular robust factored MDP, we define  $\Phi$  as, for any transition kernel  $P(s'|s,a) = \prod_{i=1}^{d} P_i(s'[i]|s[\mathrm{pa}_i], a) \in \mathcal{P}_{\mathrm{M}}$ ,  $\Phi(P) = \bigotimes_{(s,a) \in \mathcal{S} \times \mathcal{A}} \mathcal{P}_{\mathrm{Fac},\rho}(s,a;P)$ , with
$$
\mathcal {P} _ {\mathrm {F a c}, \rho} (s, a; P) = \left\{\prod_ {i = 1} ^ {d} \widetilde {P} _ {i} (\cdot): \widetilde {P} _ {i} (\cdot) \in \Delta (\mathcal {O}), D \big (\widetilde {P} _ {i} (\cdot) \| P _ {i} (\cdot | s [ \mathrm {p a} _ {i} ], a) \big) \leq \rho_ {i}, \forall i \in [ d ] \right\}.
$$
for some (pseudo-)distance  $D$  on  $\Delta (\mathcal{O})$  and some positive real numbers  $\{\rho_i\}_{i = 1}^d$
2.3 Offline Reinforcement Learning in Robust Markov Decision Processes
In this section, we define the offline RL protocol in a RMDP  $(S, \mathcal{A}, H, P^{\star}, R, \mathcal{P}_{\mathrm{M}}, \Phi)$ . The learner is given the realizable model space  $\mathcal{P}_{\mathrm{M}}$  and the robust mapping  $\Phi$ , but the learner doesn't know the transition kernel  $P^{\star}$ . For simplicity, we assume that the learner knows the reward function  $R^2$ .
Offline dataset generation. We assume that the learner is given an offline dataset  $\mathbb{D}$  that consists of  $n$  i.i.d. trajectories generated from the standard MDP  $(S, \mathcal{A}, H, P^{\star}, R)$  using some behavior policy  $\pi^{\mathrm{b}}$ . For each  $\tau \in [n]$ , the trajectory has the form of  $\{(s_h^\tau, a_h^\tau, r_h^\tau)\}_{h=1}^H$ , satisfying that  $a_h^\tau \sim \pi_h^{\mathrm{b}}(\cdot | s_h^\tau)$ ,  $r_h^\tau = R_h(s_h^\tau, a_h^\tau)$ , and  $s_{h+1}^\tau \sim P_h^\star(\cdot | s_h^\tau, a_h^\tau)$  for each step  $h \in [H]$ .
Given transition kernels  $P = \{P_h\}_{h=1}^H$  and a policy  $\pi$ , we use  $d_{P,h}^{\pi}(\cdot, \cdot)$  to denote the state-action visitation distribution at step  $h$  when following policy  $\pi$  and transition kernel  $P$ . With this notation, the distribution of  $(s_h^\tau, a_h^\tau)$  can be written as  $d_{P^*,h}^{\pi^{\mathrm{b}}}$  or simply  $d_{P^*,h}^{\mathrm{b}}$ , for each  $\tau \in [n]$  and  $h \in [H]$ . We also use  $d_{P^*,h}^{\pi^{\mathrm{b}}}(\cdot)$  to denote the marginal distribution of state at step  $h$  when there is no confusion.
Learning objective. In offline robust RL, the goal is to learn the policy  $\pi^{\star}$  from the offline dataset  $\mathbb{D}$  which maximizes the robust value function  $V_{1,P^{\star},\Phi}^{\pi}$ , that is,
$$
\pi^ {\star} = \underset {\pi \in \Pi} {\operatorname {a r g s u p}} V _ {1, P ^ {\star}, \Phi} ^ {\pi} (s _ {1}), \quad s _ {1} \in \mathcal {S}, \tag {2.10}
$$
where  $\Pi = \{\pi = \{\pi_h\}_{h=1}^H \mid \pi_h : \mathcal{S} \mapsto \Delta(\mathcal{A})\}$  denotes the collection of all Markovian policies. In view of (2.10), we call  $\pi^\star$  the optimal robust policy. Equivalently, we want to learn a policy  $\widehat{\pi} \in \Pi$  which minimizes the suboptimality gap between  $\widehat{\pi}$  and  $\pi^\star$ , defined as<sup>3</sup>
$$
\operatorname {S u b O p t} \left(\widehat {\pi}; s _ {1}\right) := V _ {1, P ^ {*}, \Phi} ^ {\pi^ {*}} \left(s _ {1}\right) - V _ {1, P ^ {*}, \Phi} ^ {\widehat {\pi}} \left(s _ {1}\right), \quad \forall s _ {1} \in \mathcal {S}. \tag {2.11}
$$
2This is reasonable since learning the reward function is easier than learning the transition kernel.
Without loss of generality, we assume that the initial state is fixed to some  $s_1 \in S$ . Our algorithm and theory can be directly extended to the case when  $s_1 \sim \rho \in \Delta(S)$ .
Algorithm 1 Doubly Pessimistic Model-based Policy Optimization (P²MP0)
1: Input: model space  $\mathcal{P}_{\mathrm{M}}$  mapping  $\Phi$  dataset  $\mathbb{D}$  policy class  $\Pi$  algorithm ModelEst.   
2: Model estimation step:   
3: Obtain a confidence region  $\widehat{\mathcal{P}} = \mathrm{ModelEst}(\mathbb{D},\mathcal{P}_{\mathrm{M}})$    
4: Doubly pessimistic policy optimization step:   
5: Set policy  $\widehat{\pi}$  as  $\begin{array}{r}\arg \sup_{\pi \in \Pi}J_{\mathrm{Pess}^2}(\pi), \end{array}$  where  $J_{\mathrm{Pess}^2}(\pi)$  is defined in (3.1).   
6: Output:  $\widehat{\pi} = \{\widehat{\pi}_h\}_{h = 1}^H$
3 Algorithm: Generic Framework and Unified Theory
In this section, we propose Doubly Pessimistic Model-based Policy Optimization  $(\mathbb{P}^2\mathbb{M}\mathbb{P}0)$  algorithm to solve offline RL in the RMDP framework we introduce in Section 2.1, and we establish a unified theoretical guarantee for  $\mathbb{P}^2\mathbb{M}\mathbb{P}0$ . Our proposed algorithm and theory show that double pessimism is a general principle for designing efficient algorithms for offline robust RL. The algorithm features three key points: i) learning the optimal robust policy  $\pi^{\star}$  approximately; ii) requiring only a partial coverage property of the offline dataset  $\mathbb{D}$ ; iii) able to handle infinite state space via function approximations.
We first introduce our proposed algorithm framework  $\mathbb{P}^2\mathbb{MPO}$  in Section 3.1. Then we establish a unified analysis for  $\mathbb{P}^2\mathbb{MPO}$  in Section 3.2. Our algorithm framework can be specified to solve all the concrete examples of RMDP we introduce in Section 2.2, which we show in Section 4.
3.1 Algorithm Framework:  $\mathbf{P}^2\mathbf{MPO}$
The  $\mathbb{P}^2\mathbb{MPO}$  algorithm framework (Algorithm 1) consists of a model estimation step and a doubly pessimistic policy optimization step, which we introduce in the following respectively.
Model estimation step (Line 3). The  $\mathsf{P}^2\mathsf{MPO}$  algorithm framework first constructs an estimation of the transition kernels  $P^{\star} = \{P_h^{\star}\}_{h = 1}^H$ , i.e., it estimates the dynamic of the training environment. It implements a sub-algorithm ModelEst  $(\mathbb{D},\mathcal{P}_{\mathrm{M}})$  that returns a confidence region  $\widehat{\mathcal{P}}$  for  $P^{\star} = \{P_h^\star \}_{h = 1}^H$ . Specifically,  $\widehat{\mathcal{P}} = \{\widehat{\mathcal{P}}_h\}_{h = 1}^H$  with  $\widehat{\mathcal{P}}_h\subseteq \mathcal{P}_{\mathrm{M}}$  for each step  $h\in [H]$ .
The sub-algorithm ModelEst can be tailored to specific RMDPs. We refer to Section 4 for detailed implementations of ModelEst for different examples of RMDPs introduced in Section 2.2. Ideally, to ensure sample-efficient learning, we need  $\widehat{\mathcal{P}} = \mathrm{ModelEst}(\mathbb{D},\mathcal{P}_{\mathrm{M}})$  to satisfy: i) the transition kernels  $P^{\star} = \{P_{h}\}_{h = 1}^{H}$  are contained in  $\widehat{\mathcal{P}} = \{\widehat{\mathcal{P}}_h\}_{h = 1}^H$  ; ii) each transition kernel  $P_{h}\in \widehat{\mathcal{P}}_{h}$  enjoys a small "robust estimation error" which is highly related to the robust Bellman equation in (2.5). We quantify these two conditions of  $\widehat{\mathcal{P}}$  for sample-efficient learning in Section 3.2.
Doubly pessimistic policy optimization step (Line 5). After model estimation step,  $\mathsf{P}^2\mathsf{MPO}$  performs policy optimization to find the optimal robust policy. To learn the optimal robust policy in the face of uncertainty,  $\mathsf{P}^2\mathsf{MPO}$  adopts a double pessimism principle. To explain, this general principle has two sources of pessimism: i) pessimism in the face of data uncertainty; ii) pessimism to find a robust policy. Specifically, for any policy, we first estimate its robust value function via two infimums, where one is an infimum over the confidence set constructed in the model estimation step, and one is an infimum over the robust sets. Formally, for any  $\pi \in \Pi$ , we define the doubly pessimistic estimator
$$
J _ {\mathrm {P e s s} ^ {2}} (\pi) = \inf  _ {P _ {h} \in \widehat {\mathcal {P}} _ {h}, 1 \leq h \leq H} \inf  _ {\widetilde {P} _ {h} \in \Phi (P _ {h}), 1 \leq h \leq H} V _ {1} ^ {\pi} \left(s _ {1}; \{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}\right), \tag {3.1}
$$
where  $V_{1}^{\pi}$  is the standard value function of policy  $\pi$  defined in (2.3). Then  $\mathbb{P}^2\mathbb{MPO}$  outputs the policy  $\widehat{\pi}$  that maximizes the doubly pessimistic estimator  $J_{\mathrm{Pess}^2}(\pi)$  defined in (3.1), i.e.,
$$
\widehat {\pi} = \underset {\pi \in \Pi} {\operatorname {a r g s u p}} J _ {\text {P e s s} ^ {2}} (\pi). \tag {3.2}
$$
The novelty of the doubly pessimistic policy optimization step is performing pessimism from the two sources (data uncertainty and robust optimization) simultaneously. Compared with the previous works on standard offline RL in MDPs [62, 54] and offline RL in RMDPs without pessimism [68, 77, 40], they only contain one source of pessimism in algorithm design, contrasting with our algorithm.
We note that a recent work [47] also studied robust offline RL in  $S \times \mathcal{A}$ -rectangular robust tabular MDPs (Example 2.5) using pessimism techniques. Compared with our double pessimism principle,
their algorithm performs pessimism in face of data uncertainty i) depending on the tabular structure of the model since a point-wise pessimism penalty is needed and ii) depending on the specific form of the robust set  $\Phi(P)$ , which makes it difficult to adapt to the infinite state space case with general function approximations and general types of robust set  $\Phi(P)$ .
3.2 Unified Theoretical Analysis
In this section, we establish a unified theoretical analysis for the  $\mathrm{P}^2\mathrm{MP0}$  algorithm framework proposed in Section 3.1. We first specify the two conditions that the model estimation step of  $\mathrm{P}^2\mathrm{MP0}$  should satisfy in order for sample-efficient learning. Then we establish an upper bound of suboptimality of the policy obtained by  $\mathrm{P}^2\mathrm{MP0}$  given that these two conditions are satisfied. In Section 4, we show that the specific implementations of the sub-algorithm ModelEst for the RMDPs examples in Section 2.2 satisfy these two conditions, which results in tailored suboptimality bounds for these examples.
Conditions. The two conditions on the model estimation step are given by the following.
Condition 3.1 ( $\delta$ -accuracy). With probability at least  $1 - \delta$ , it holds that  $P_h^\star \in \widehat{\mathcal{P}}_h$  for any  $h \in [H]$ .
Condition 3.2 ( $\delta$ -model estimation error). With probability at least  $1 - \delta$ , it holds that
$$
\mathbb {E} _ {(s, a) \sim d _ {P ^ {\star}, h} ^ {\pi^ {\mathrm {b}}}} \left[ \left(\inf _ {\widetilde {P} _ {h} \in \Phi (P _ {h})} \widetilde {\mathbb {P}} _ {h} (V _ {h + 1, P, \Phi} ^ {\pi^ {\star}}) (s, a) - \inf _ {\widetilde {P} _ {h} \in \Phi (P _ {h} ^ {\star})} \widetilde {\mathbb {P}} _ {h} (V _ {h + 1, P, \Phi} ^ {\pi^ {\star}}) (s, a)\right) ^ {2} \right] \leq \operatorname {E r r} _ {h} ^ {\Phi} (n, \delta).
$$
for any  $P = \{P_h\}_{h=1}^H$  with  $P_h \in \widehat{\mathcal{P}}_h$ . Here  $\widetilde{\mathbb{P}}_h(V_{h+1,P,\Phi}^{\pi^\star})(s,a) = \mathbb{E}_{s' \sim \widetilde{P}_h(\cdot|s,a)}[V_{h+1,P,\Phi}^{\pi^\star}(s')]$ .
Condition 3.1 requires that the confidence region  $\widehat{\mathcal{P}}_h$  contains the transition kernel of the training environment  $P_h^\star$  with high probability. Condition 3.2 requires that each transition kernel  $P_h\in \widehat{\mathcal{P}}_h$  induces an error from  $P_{h}^{\star}$  less than certain quantity  $\mathrm{Err}_h^\Phi (n,\delta)$ , where the error is adapted from the robust Bellman equation (2.5) and involves an infimum over the robust set of  $P_{h}$  and  $P_{h}^{\star}$ . In specific implementations of ModelEst for RMDP examples in Section 4, we show that the quantity  $\mathrm{Err}_h^\Phi (n,\delta)$  generally scales with  $\widetilde{\mathcal{O}} (n^{-1})$ , where  $n$  is the number of trajectories in the offline dataset.
Suboptimality analysis. Now we establish a unified suboptimality bound for the  $\mathrm{P}^2\mathrm{MP0}$  algorithm framework. Thanks to the double pessimism principle of  $\mathrm{P}^2\mathrm{MP0}$ , we can prove a suboptimality bound while only making a mild robust partial coverage assumption on the dataset.
Assumption 3.3 (Robust partial coverage). We assume that
$$
C _ {P ^ {\star}, \Phi} ^ {\star} := \sup  _ {1 \leq h \leq H} \sup  _ {P = \{P _ {h} \} _ {h = 1} ^ {H}, P _ {h} \in \Phi (P _ {h} ^ {\star})} \mathbb {E} _ {(s, a) \sim d _ {P ^ {\star}, h} ^ {\pi^ {\mathrm {b}}}} \left[ \left(\frac {d _ {P , h} ^ {\pi^ {\star}} (s , a)}{d _ {P ^ {\star} , h} ^ {\pi^ {\mathrm {b}}} (s , a)}\right) ^ {2} \right] <   + \infty ,
$$
and we call  $C_{P^*,\Phi}^\star$  the robust partial coverage coefficient.
To interpret, Assumption 3.3 only requires that the dataset covers the visitation distribution of the optimal policy  $\pi^{\star}$ , but in a robust fashion since  $C_{P^{\star},\Phi}^{\star}$  considers all possible transition kernels in the robust set  $\Phi(P^{\star})$ . The robust consideration in  $C_{P^{\star},\Phi}^{\star}$  is because in RMDPs the policies are all evaluated in a robust way. This partial-coverage-style assumption is much weaker than full-coverage-style assumptions [68, 77, 40] which require either a uniformly lower bounded dataset distribution or covering the visitation distribution of any  $\pi \in \Pi$ . For  $S \times \mathcal{A}$ -rectangular robust tabular MDPs (Example 2.5), the robust partial coverage coefficient  $C_{P^{\star},\Phi}^{\star}$  is similar with the partial coverage coefficient proposed by [47] who studied tabular RMDPs under partial coverage. We highlight that beyond  $S \times \mathcal{A}$ -rectangular robust tabular MDPs, our robust partial coverage assumption can handle other examples of RMDPs (Section 2.2) under our unified theory.
Our main result is the following theorem. See Appendix C for a detailed proof.
Theorem 3.4 (Suboptimality of  $\mathbb{P}^2\mathbb{MPO}$ ). Under Assumptions 2.2 and 3.3, suppose that Algorithm 1 implements a sub-algorithm that satisfies Conditions 3.1 and 3.2, then with probability at least  $1 - 2\delta$
$$
\operatorname {S u b O p t} (\widehat {\pi}; s _ {1}) \leq \sqrt {C _ {P ^ {\star} , \Phi} ^ {\star}} \cdot \sum_ {h = 1} ^ {H} \sqrt {\operatorname {E r r} _ {h} ^ {\Phi} (n , \delta)}.
$$
When  $\mathrm{Err}_h^\Phi (n,\delta)$  achieves a rate of  $\widetilde{\mathcal{O}} (n^{-1})$ , then  $\mathbb{P}^2\mathbb{M}\mathbb{P}\mathbb{O}$  enjoys a  $\widetilde{\mathcal{O}} (n^{-1 / 2})$ -suboptimality. In the following Section 4, we give specific implementations of the model estimation step of  $\mathbb{P}^2\mathbb{M}\mathbb{P}\mathbb{O}$  for each example of RMDP in Section 2. The implementations will make Conditions 3.1 and 3.2 satisfied and thus specify the unified result Theorem 3.4.
4 Implementations of  $\mathbf{P}^2\mathbf{MPO}$  for Examples of RMDPs
In this section, we provide concrete implementations of the ModelEst sub-algorithm in  $\mathsf{P}^2\mathsf{MP0}$  (Algorithm 1). In Section 4.1, we implement ModelEst for all the RMDPs that satisfy Assumption 2.2, and we specify the suboptimality bounds in Theorem 3.4 to Examples 2.5, 2.7, 2.8 in Section 2.2. In Section 4.2, we implement ModelEst for  $S \times \mathcal{A}$ -rectangular robust factored MDPs (Example 2.9) and specify Theorem 3.4 to this example.
4.1 Model Estimation for General RMDPs with  $S \times \mathcal{A}$ -rectangular Robust Sets
Using the offline data  $\mathbb{D}$ , we first construct the maximum likelihood estimator (MLE) of the transition kernel  $P^{\star}$ . Specifically, for each step  $h\in [H]$ , we define
$$
\widehat {P} _ {h} = \underset {P \in \mathcal {P} _ {\mathrm {M}}} {\arg \max } \frac {1}{n} \sum_ {\tau = 1} ^ {n} \log P \left(s _ {h + 1} ^ {\tau} \mid s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right). \tag {4.1}
$$
After, we construct a confidence region for the MLE estimator, denoted by  $\widehat{\mathcal{P}}$ . Specifically,  $\widehat{\mathcal{P}}$  contains all transitions which have a small total variance distance from  $\widehat{P}$ . For each step  $h \in [H]$ , we define
$$
\widehat {\mathcal {P}} _ {h} = \left\{P \in \mathcal {P} _ {\mathrm {M}}: \frac {1}{n} \sum_ {\tau = 1} ^ {n} \| \widehat {P} _ {h} \left(\cdot \mid s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) - P \left(\cdot \mid s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \| _ {1} ^ {2} \leq \xi \right\}. \tag {4.2}
$$
Here  $\xi > 0$  is a tuning parameter that controls the size of the confidence region  $\widehat{\mathcal{P}}_h$ . Finally, we set ModelEst  $(\mathbb{D}, \mathcal{P}_{\mathrm{M}}) = \widehat{\mathcal{P}} = \{\widehat{\mathcal{P}}_h\}_{h=1}^H$  with  $\widehat{\mathcal{P}}_h$  given in (4.2). In the sequel, we mainly consider the distance  $D(\cdot \| \cdot)$  in Assumption 2.2 to be KL-divergence and TV-distance. The following corollary specifies Theorem 3.4 to model estimation step given by (4.2). See Appendix D for a detailed proof.
Corollary 4.1 (Suboptimality of  $\mathbb{P}^2\mathbb{M}\mathbb{P}0$ :  $S\times \mathcal{A}$ -rectangular robust MDP). Under Assumption 2.2, 3.3, setting the tuning parameter  $\xi$  as
$$
\xi = \frac {C _ {1} \log (C _ {2} H \mathcal {N} _ {\perp} (1 / n ^ {2} , \mathcal {P} _ {\mathrm {M}} , \| \cdot \| _ {1 , \infty}) / \delta)}{n},
$$
for some constants  $C_1, C_2 > 0$ ,  $P^2 MPO$  with model estimation step given by (4.2) satisfies that
$\spadesuit$  when  $D(\cdot \|\cdot)$  is KL-divergence and Assumption D.3 holds with parameter  $\underline{\lambda}$ , then with probability at least  $1 - 2\delta$ ,
$$
\mathrm {S u b O p t} (\widehat {\pi}; s _ {1}) \leq \frac {\sqrt {C _ {P ^ {*} , \Phi} ^ {\star}} H ^ {2} \exp (H / \underline {{\lambda}})}{\rho} \cdot \sqrt {\frac {C _ {1} ^ {\prime} \log (C _ {2} ^ {\prime} H \mathcal {N} _ {[ ]} (1 / n ^ {2} , \mathcal {P} _ {\mathrm {M}} , \| \cdot \| _ {1 , \infty}) / \delta)}{n}}.
$$
$\spadesuit$  when  $D(\cdot \|\cdot)$  is TV-divergence, then with probability at least  $1 - 2\delta$ ,
$$
\mathrm {S u b O p t} (\widehat {\pi}; s _ {1}) \leq \sqrt {C _ {P ^ {\star} , \Phi} ^ {\star}} H ^ {2} \cdot \sqrt {\frac {C _ {1} ^ {\prime} \log (C _ {2} ^ {\prime} H \mathcal {N} _ {[ ]} (1 / n ^ {2} , \mathcal {P} _ {\mathrm {M}} , \| \cdot \| _ {1 , \infty}) / \delta)}{n}}.
$$
$\mathcal{S} \times \mathcal{A}$ -rectangular robust tabular MDP (Example 2.5). When  $\mathcal{S}$  is finite as in Example 2.5, the MLE estimator (4.1) coincides with the empirical estimator
$$
\widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right) = \frac {\sum_ {\tau = 1} ^ {n} \mathbf {1} \left\{s _ {h} ^ {\tau} = s , a _ {h} ^ {\tau} = a , s _ {h + 1} ^ {\tau} = s ^ {\prime} \right\}}{1 \vee \sum_ {\tau = 1} ^ {n} \mathbf {1} \left\{s _ {h} ^ {\tau} = s , a _ {h} ^ {\tau} = a \right\}}, \tag {4.3}
$$
which is adopted by [77, 68, 39, 47, 40]. Furthermore, in Example 2.5, the realizable model space  $\mathcal{P}_{\mathrm{M}} = \{P:\mathcal{S}\times \mathcal{A}\mapsto \Delta (\mathcal{S})\}$ . When  $\mathcal{S}$  and  $\mathcal{A}$  are finite, we can bound the bracket number of  $\mathcal{P}_{\mathrm{M}}$  as
$$
\log \mathcal {N} _ {\llbracket} \left(1 / n ^ {2}, \mathcal {P} _ {\mathrm {M}}, \| \cdot \| _ {1, \infty}\right) \leq 2 | \mathcal {S} | ^ {2} | \mathcal {A} | \log (n). \tag {4.4}
$$
Combining (4.4) and Corollary 4.1, we can conclude that: i) under TV-distance the suboptimality of  $\mathbb{P}^2\mathbb{MPO}$  for  $S\times \mathcal{A}$ -rectangular robust tabular RMDP is given by  $\mathcal{O}(H^{2}\sqrt{C_{P^{\star},\Phi}^{\star}|S|^{2}|\mathcal{A}|\log(nH / \delta) / n})$ , ii) under KL-divergence the suboptimality of  $\mathbb{P}^2\mathbb{MPO}$  for  $S\times \mathcal{A}$ -rectangular robust tabular MDP is given by  $\mathcal{O}(H^{2}\exp(H / \underline{\lambda}) / \rho \cdot \sqrt{C_{P^{\star},\Phi}^{\star}|S|^{2}|\mathcal{A}|\log(nH / \delta) / n})$ . We prove (4.4) in Appendix D.2.