SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 4.2. We note that for KL-divergence robust sets, the dependence on  $\exp (H)$  is due to the usage of general function approximations, which also appears in a recent work [29] for RMDPs with linear function approximations. For the special case of robust tabular MDPs, under KL-divergence, existing work [47] derived sample complexities without  $\exp (H)$ , but with an additional dependence on  $1 / d_{\min}^{\mathrm{b}}$  and  $1 / P_{\min}^{\star}$ . Here  $d_{\min}^{\mathrm{b}} = \min_{(s,a,h):d_{P^{\star},h}^{\pi^{\mathrm{b}}} (s,a) > 0}d_{P^{\star},h}^{\pi^{\mathrm{b}}}(s,a)$  and  $P_{\min}^{\star} = \min_{(s,s',h):P_h(s'|s,\pi_h^\star (s)) > 0}P_h^\star (s'|s,\pi_h^\star (s))$ . We remark that our analysis for  $\mathsf{P}^{2}\mathsf{MP0}$  algorithm can be tailored to the tabular case and become  $\exp (H)$ -free using their techniques, with the cost of an additional dependence on  $1 / d_{\min}^{\mathrm{b}}$  and  $1 / P_{\min}^{\star}$ . But we note that in the infinite state space case, both the  $1 / d_{\min}^{\mathrm{b}}$ -dependence and the  $1 / P_{\min}$ -dependence become problematic. So, it serves as an interesting future work to answer whether one can derive both  $\exp (H)$ -free and  $(1 / d_{\min}^{\mathrm{b}},1 / P_{\min}^{\star})$ -free results for (general) function approximations under KL-divergence.
$\mathcal{S} \times \mathcal{A}$ -rectangular robust MDP with kernel and neural function approximations (Examples 2.7 and 2.8). By specifying the bracket numbers in Corollary 4.1, we can provide the detailed suboptimality guarantees for  $\mathcal{S} \times \mathcal{A}$ -rectangular robust MDP with kernel and neural function approximations. Due to space limitations, we defer the detailed results to Appendices D.3 and D.4.
4.2 Model Estimation for  $S \times \mathcal{A}$ -rectangular Robust Factored MDPs (Example 2.9)
We first construct MLE estimator for each factor  $P_{h,i}^{\star}$  of the transition  $P_{h}^{\star} = \prod_{i=1}^{d} P_{h,i}^{\star}$ , that is,
$$
\widehat {P} _ {h, i} = \underset {P _ {i}: S [ \mathrm {p a} _ {i} ] \times \mathcal {A} \mapsto \Delta (\mathcal {O})} {\arg \max } \frac {1}{n} \sum_ {k = 1} ^ {n} \log P \left(s _ {h + 1} ^ {\tau} [ i ] \mid s _ {h} ^ {\tau} [ \mathrm {p a} _ {i} ], a _ {h} ^ {\tau}\right). \tag {4.5}
$$
Then given  $\{\widehat{P}_{h,i}\}_{i = 1}^{d}$  we construct a confidence region that is factored across  $i\in [d]$ . Specifically,
$$
\widehat {\mathcal {P}} _ {h} = \left\{P \left(s ^ {\prime} \mid s, a\right) = \prod_ {i = 1} ^ {d} P _ {i} \left(s ^ {\prime} [ i ] \mid s [ \mathrm {p a} _ {i} ], a\right): \frac {1}{n} \sum_ {i = 1} ^ {n} \| \left(P _ {i} - \widehat {P} _ {h, i}\right) \left(\cdot \mid s _ {h} ^ {\tau} [ \mathrm {p a} _ {i} ], a _ {h} ^ {\tau}\right) \| _ {1} ^ {2} \leq \xi_ {i}, \forall i \right\}. \tag {4.6}
$$
Finally, we set ModelEst  $(\mathbb{D},\mathcal{P}_{\mathrm{M}}) = \widehat{\mathcal{P}} = \{\widehat{\mathcal{P}}\}_{h = 1}^{H}$  with  $\widehat{\mathcal{P}}_h$  given in (4.6). The following corollary specifies Theorem 3.4 to model estimation step given by (4.6). See Appendix E for a detailed proof. Corollary 4.3 (Suboptimality of  $\mathsf{P}^2\mathsf{MP0}$ :  $S\times \mathcal{A}$ -rectangular robust factored MDP). Supposing the RMDP is an  $S\times \mathcal{A}$ -rectangular robust factored MDP, under the same Assumptions and parameter choice in Theorem 3.4 and Proposition E.1,  $P^2\mathsf{MP0}$  with model estimation step given by (4.6) satisfies
$\clubsuit$  when  $D(\cdot \|\cdot)$  is KL-divergence and Assumption D.3 holds with parameter  $\underline{\lambda}$ , then with probability at least  $1 - 2\delta$ , (defining  $\rho_{\mathrm{min}} = \min_{i \in [d]} \rho_i$ )
$$
\mathrm {S u b O p t} (\widehat {\pi}; s _ {1}) \leq \frac {\sqrt {C _ {P ^ {*} , \Phi} ^ {\star}} H ^ {2} \exp (H / \underline {{\lambda}})}{\rho_ {\mathrm {m i n}}} \cdot \sqrt {\frac {d C _ {1} ^ {\prime} \sum_ {i = 1} ^ {d} | \mathcal {O} | ^ {1 + | \mathrm {p a} _ {i} |} | \mathcal {A} | \log (C _ {2} ^ {\prime} n d / \delta)}{n}}.
$$
$\clubsuit$  when  $D(\cdot \|\cdot)$  is TV-divergence, then with probability at least  $1 - 2\delta$ ,
$$
\operatorname {S u b O p t} (\widehat {\pi}; s _ {1}) \leq \sqrt {C _ {P ^ {\star} , \Phi} ^ {\star}} H ^ {2} \sqrt {\frac {d C _ {1} ^ {\prime} \sum_ {i = 1} ^ {d} | \mathcal {O} | ^ {1 + | \mathrm {p a} _ {i} |} | \mathcal {A} | \log \left(C _ {2} ^ {\prime} n d / \delta\right)}{n}}.
$$
Compared with the suboptimality bounds for  $S \times \mathcal{A}$ -rectangular robust MDPs in Section 4.1, the suboptimality of  $S \times \mathcal{A}$ -rectangular robust factored MDPs with ModelEst given in (4.6) only scales with  $\sum_{i=1}^{d} |\mathcal{O}|^{1 + |\mathrm{pa}_i|}$  instead of scaling with  $|S| = \prod_{i=1}^{d} |\mathcal{O}|$  which is of order  $\exp(d)$ . This justifies the benefit of considering  $S \times \mathcal{A}$ -rectangular robust factored MDPs when the transition kernels of training and testing environments enjoy factored structures.
This paper proposes a general learning principle — double pessimism — for robust offline RL. Based on this learning principle, we propose a generic algorithm that only requires robust partial coverage data to solve  $S \times \mathcal{A}$ -rectangular RMDPs with general function approximation. Our results are ready to be extended to  $d$ -rectangular linear RMDPs [29]. See Appendix A for details. In Appendix A, we also provide some challenges to perform sample efficient RL in  $S$ -rectangular RMDPs.
The material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397. Additional support is gratefully acknowledged from NSF 1915967, 2118199, 2229012, 2312204.
[1] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
[2] András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71(1):89-129, 2008.
[3] Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pages 511-520. PMLR, 2021.
[4] Qi Cai, Zhuoran Yang, Csaba Szepesvari, and Zhaoran Wang. Optimistic policy optimization with general function approximations. 2020.
[5] Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019.
[6] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarily trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.02446, 2022.
[7] Pierre Clavier, Erwan Le Pennec, and Matthieu Geist. Towards minimax optimality of model-based robust reinforcement learning. arXiv preprint arXiv:2302.05372, 2023.
[8] Jing Dong, Jingwei Li, Baoxiang Wang, and Jingzhao Zhang. Online policy optimization for robust mdp. arXiv preprint arXiv:2209.13841, 2022.
[9] Laurent El Ghaoui and Arnab Nilim. Robust solutions to markov decision problems with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
[10] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Robust  $\phi$ -divergence mdps. 2022.
[11] Jiachen Hu, Han Zhong, Chi Jin, and Liwei Wang. Provable sim-to-real transfer in continuous domain with partial observations. arXiv preprint arXiv:2210.15598, 2022.
[12] Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, pages 1695-1724, 2013.
[13] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
[14] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[15] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143. PMLR, 2020.
[16] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[17] Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pages 740-747, 1999.
[18] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-1274, 2013.
[71] Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34:4065-4078, 2021.
[72] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
[73] Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626-13640, 2021.
[74] Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730–2775. PMLR, 2022.
[75] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and ChoiJui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024-21037, 2020.
[76] Wenshuai Zhao, Jorge Pña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 737-744. IEEE, 2020.
[77] Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331-3339. PMLR, 2021.