Title: Near-Optimal Sample Complexities of  Divergence-based  S-rectangular Distributionally Robust Reinforcement Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: distributionally robustness

reinforcement learning

markov decision process
Abstract: Distributionally robust reinforcement learning (DR-RL) has recently gained significant attention as a principled approach that addresses discrepancies between training and testing environments. To balance robustness, conservatism, and computational traceability, the literature has introduced DR-RL models with SA-rectangular and S-rectangular adversaries. While most existing statistical analyses focus on SA-rectangular models, owing to their algorithmic simplicity and the optimality of deterministic policies, S-rectangular models more accurately capture distributional discrepancies in many real-world applications and often yield more effective robust randomized policies. In this paper, we study the empirical value iteration algorithm for divergence-based S-rectangular DR-RL and establish near-optimal sample complexity bounds of $\widetilde{O}(|\mathcal{S}||\mathcal{A}|(1-\gamma)^{-4}\varepsilon^{-2})$, where $\varepsilon$ is the target accuracy, $|\mathcal{S}|$ and $|\mathcal{A}|$ denote the cardinalities of the state and action spaces, and $\gamma$ is the discount factor. To the best of our knowledge, these are the first sample complexity results for divergence-based S-rectangular models that achieve optimal dependence on $|\mathcal{S}|$, $|\mathcal{A}|$, and $\varepsilon$ simultaneously. We further validate this theoretical dependence through numerical experiments on a robust inventory control problem and a theoretical worst-case example, demonstrating the fast learning performance of our proposed algorithm.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work obtains the first near-optimal sample complexity bound $\\widetilde{O}[|S||A|(1-\\gamma)^{-4}\\epsilon^{-2}]$ of empirical value iteration algorithm on distributionally robust reinforcement learning (DR-RL) problem with $f$-divergence based $s$-rectangular uncertainty set.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This is a complete and clear work. The topic of distributionally robust reinforcement learning is popular with many applications. It achieves the first theoretical sample complexity result with optimal dependence on $\\epsilon,|S|,|A|$, and the complexity result is verified by simulations. The result looks reasonable to me based on my knowledge of value iteration and sample-based transition kernel estimation. The proof has improved the existing techniques from [25]. This work may be cited in the future in either of two ways: (1) Apply this algorithm to small state-action spaces due to efficient sample complexity obtained by this work. (2) Improve this work by future works for more practical and tricky settings such as large state-action spaces, lack of generative model, model-free setting, offline reinforcement learning, etc. \n\nWeaknesses: The second experiment in Section 5.2 needs more clarification to be reproducible (See my questions below). It also would be better to compare with other existing sample-based value iteration algorithms in the experiments. A few more citations are needed (See my questions below). There are some limitations as listed in the conclusion section.\n\nQUESTIONS:\n(1) What's the theoretical complexity lower bound? You could write it down and cite the corresponding work, in the introduction and other proper places. The conclusion section mentions the complexity gap as a limitation. Does it mean dependence on $1-\\gamma$ is not optimal? You might make it more explicit. \n\n(2) Could you cite the following works of value iteration on robust MDP? Also, [1] seems to have better dependence on $\\epsilon$. How to explain? \n\n[1] Grand-Clément, J. and Kroer, C. (2021). Scalable first-order methods for robust mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12086–12094.\n\n[2] Kumar, N., Levy, K., Wang, K., and Mannor, S. (2023b). An efficient solution to s-rectangular robust markov decision processes. ArXiv:2301.13642.\n\n(3) Are Lemmas 1 and 2 proposed by you or existing literature? If latter, please cite.\n\n(4) To reproduce Section 5.2 (the second experiment), what does \"estimation error\" mean? (might be defined by equations) What is the discount factor $\\gamma$? \n\nMinor issues:\n\n(5) $A_t$ instead of $a_t$ in Examples 1. \n\n(6) $\\|R+\\gamma v\\| _ {\\infty}$ in Proposition 2 could be defined, especially considering that $R(s,a)$ and $v(s)$ take different input variables. \n\n(7) In Section 5, does $n_0=n$? \n\n(8) Figure 4 caption: \"number of actions\".\n\nLIMITATIONS:\nYes, the authors have written their two limitations in the conclusion section: (1) \"The reliance on access to a generative model and the presence of a gap between our upper bound and the minimax lower bound established in Yang [29]\"; (2) \"As part of future work, we aim to develop provable theoretical guarantees for other settings, including model-free algorithms and offline reinforcement learning.\"", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes sample complexity of value iteration for divergence based s-rectangular robust MDPs. It assumes the generative model of the MDP, and constructs empirical nominal kernels using the samples, then use this for Bellman updates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Established $O(\\epsilon^{-2})$ sample complexity of the divergence based robust MDPs.\n\nWeakness: 1) The approach builds empirical kernel using samples which is not feasible for large states space MDPs. 2) Missing large body of relevant works, see questions.\n\nQUESTIONS:\n***Q1)  Para (Line 40-44) of the paper discusses the nature of the optimal policies for SA/S -rect. Robust MDPs. The should mention that the optimal policies for s-rect can be stochastic [1] and a exact nature (threshold power policy) was shown in [2] for L_p bounded s-rect MDP.***\n\n***Q2) Literature review missing mention large bodies of relevant works such as s-rectangular works [2,4,8] ,  non -rectangular works  [3,5] that are even less convervative than s-rect,  KL-based SA-rect work [6], and [7] that has sample complexity for R-contamination uncertainty sets***\n\n***Q3) Please compare your techniques and results with [9,10], it also claims $O(\\epsilon^{-2})$ sample complexity for policy evaluation  diverence based robust MDPs. Can policy evaluation be trivially used to get optimal value funtion using policy iteration ?***\n\n[1] Wolfram Wiesemann, Daniel Kuhn, Berç Rustem, (2012) Robust Markov Decision Processes. Mathematics of Operations Research 38(1):153-183.\nhttps://doi.org/10.1287/moor.1120.0566 \n\n[2]@inproceedings{\nkumar2024efficient,\ntitle={Efficient Value Iteration for s-rectangular Robust Markov Decision Processes},\nauthor={Navdeep Kumar and Kaixin Wang and Kfir Yehuda Levy and Shie Mannor},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=J4LTDgwAZq}\n}\n\n[3] Gadot, U., Derman, E., Kumar, N., Elfatihi, M. M., Levy, K., & Mannor, S. (2024). Solving Non-rectangular Reward-Robust MDPs via Frequency Regularization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 21090-21098. https://doi.org/10.1609/aaai.v38i19.30101\n\n[4]@inproceedings{10.5555/3666122.3668721,\nauthor = {Kumar, Navdeep and Derman, Esther and Geist, Matthieu and Levy, Kfir and Mannor, Shie},\ntitle = {Policy gradient for rectangular robust Markov decision processes},\nyear = {2023},\npublisher = {Curran Associates Inc.},\naddress = {Red Hook, NY, USA},\nbooktitle = {Proceedings of the 37th International Conference on Neural Information Processing Systems},\narticleno = {2599},\nnumpages = {25},\nlocation = {New Orleans, LA, USA},\nseries = {NIPS '23}\n}\n\n[5]@misc{kumar2025dualformulationnonrectangularlp,\n      title={Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processes}, \n      author={Navdeep Kumar and Adarsh Gupta and Maxence Mohamed Elfatihi and Giorgia Ramponi and Kfir Yehuda Levy and Shie Mannor},\n      year={2025},\n      eprint={2502.09432},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2502.09432}, \n}\n\n[6] @inproceedings{\ngadot2024bring,\ntitle={Bring Your Own (Non-Robust) Algorithm to Solve Robust {MDP}s by Estimating The Worst Kernel},\nauthor={Uri Gadot and Kaixin Wang and Navdeep Kumar and Kfir Yehuda Levy and Shie Mannor},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=UqoG0YRfQx}\n}\n\n[7] @inproceedings{\nwang2021online,\ntitle={Online Robust Reinforcement Learning with Model Uncertainty},\nauthor={Yue Wang and Shaofeng Zou},\nbooktitle={Advances in Neural Information Processing Systems},\neditor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},\nyear={2021},\nurl={https://openreview.net/forum?id=IhiU6AJYpDs}\n}\n\n[8]@misc{derman2021twiceregularizedmdpsequivalence,\n      title={Twice regularized MDPs and the equivalence between robustness and regularization}, \n      author={Esther Derman and Matthieu Geist and Shie Mannor},\n      year={2021},\n      eprint={2110.06267},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2110.06267}, \n}\n\n[9]@misc{xu2025finitesampleanalysispolicyevaluation,\n      title={Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning}, \n      author={Yang Xu and Washim Uddin Mondal and Vaneet Aggarwal},\n      year={2025},\n      eprint={2502.16816},\n      archivePrefix={arXiv},\n      primaryClass={stat.ML},\n      url={https://arxiv.org/abs/2502.16816}, \n}\n\n[10]@misc{xu2025efficientqlearningactorcriticmethods,\n      title={Efficient $Q$-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learning}, \n      author={Yang Xu and Swetha Ganesh and Vaneet Aggarwal},\n      year={2025},\n      eprint={2506.07040},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2506.07040}, \n}\n\nLIMITATIONS:\nOK", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses distributionally robust RL in the S-rectangular setting. This paper establishes a statistical bound on the *estimation error* for the optimal robust value. However, it does not address the *sample complexity*.\n\nSTRENGTHS AND WEAKNESSES:\n* **Strengths:**\n    1. The writing of this paper is clear,\n    2. Numerical results are presented and align with the theoretical analysis.\n\n* **Weaknesses:** \n    1. The definitions are unclear (See Question 1 below).\n    2. The involvement of the minimum support probability (Def. 5) for $f_k$-divergence uncertainty set is sub-optimal when compared to the previously known bound [Theorem 3.2(b), 1].\n\n[1] Yang, W., Zhang, L., & Zhang, Z. (2022). Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics.\n\nQUESTIONS:\n1. If I understand correctly, Theorems 1 and 2 are **not** the *sample complexity* discussed in [1, 2]. According to Proposition 1, $\\widehat{v} = \\widehat{T}^*(\\widehat{v})$ should refer to the optimal value under the **empirical DR Bellman Operators (Def.3)**, which is not the optimal robust value function defined w.r.t the true uncertainty set (i.e., $V_{\\mathcal{P}}^{\\widehat{\\pi}}$). If that is true, I believe it is inappropriate to claim your contributions as \"the first sample complexity results for divergence-based S-rectangular models\".\n2. I think the reference in Line 332 should be [30], instead of [29].\n3. Can you explain why the bounds in Theorems 1 and 2 do not involve the uncertainty radius $\\rho$?\n\n\n[1] Clavier, P., Shi, L., Le Pennec, E., Mazumdar, E., Wierman, A., & Geist, M. (2024). Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms. Advances in Neural Information Processing Systems.\n\n[2] Yang, W., Zhang, L., & Zhang, Z. (2022). Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper focuses on divergence-based S-rectangular distributionally robust reinforcement learning (DR-RL), with the core being establishing near-optimal sample complexity bounds under this framework and validating their effectiveness through theoretical analysis and numerical experiments.

STRENGTHS AND WEAKNESSES:
Strengths
-The article considers a reward function $$R(S_t, A_t, S_{t+1})$$ that depends on the "current state, current action, and next state", a structure often overlooked in existing research. 
- The article conducts numerical experiments on a robust inventory control problem and a theoretical worst-case example, verifying that the sample complexity decays at a rate of $$1/\sqrt{n}$$ with respect to the sample size n and has a linear dependence on $$|S||A|$$, which is consistent with theoretical deductions, enhancing the credibility of the conclusion.
Weaknesses：
1.The numerical experiments only run under a single set of hyperparameters without reporting error bars or statistical significance information, making it difficult to fully verify the robustness of the results.
2.The research primarily focuses on the theoretical analysis of sample complexity, with little discussion of practical implementation challenges. This narrow focus limits its guidance for real-world applications.

QUESTIONS:
Suggestions:
1.The authors should supplement additional numerical experiments to systematically investigate the impact of key hyperparameters on the estimation error and sample complexity
2.The authors should elaborate on how the established near-optimal sample complexity theory directly guides the practical implementation of S-rectangular divergence-based DR-RL algorithms.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03