Title: Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: robust Markov games

self-play

distribution shift

model uncertainty

reinforcement learning
Abstract: Multi-agent reinforcement learning (MARL), as a thriving field, explores how multiple agents independently make decisions in a shared dynamic environment. Due to environmental uncertainties, policies in MARL must remain robust to tackle the sim-to-real gap. We focus on robust two-player zero-sum Markov games (TZMGs) in offline settings, specifically on tabular robust TZMGs (RTZMGs). We propose a model-based algorithm (*RTZ-VI-LCB*) for offline RTZMGs, which is optimistic robust value iteration combined with a data-driven Bernstein-style penalty term for robust value estimation. By accounting for distribution shifts in the historical dataset, the proposed algorithm establishes near-optimal sample complexity guarantees under partial coverage and environmental uncertainty. An information-theoretic lower bound is developed to confirm the tightness of our algorithm's sample complexity, which is optimal regarding both state and action spaces. To the best of our knowledge, RTZ-VI-LCB is the first to attain this optimality, sets a new benchmark for offline RTZMGs, and is validated experimentally.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenging problem of offline robust reinforcement learning in two-player zero-sum Markov games (RTZMGs), with a focus on tabular environments under partial coverage and environmental uncertainty. The authors introduce RTZ-VI-LCB, a novel model-based algorithm combining robust value iteration with lower confidence bounds. It incorporates a Bernstein-style penalty and a two-stage subsampling procedure to handle statistical dependencies in offline datasets. The paper provides theoretical guarantees by deriving both upper and lower bounds on sample complexity, showing that RTZ-VI-LCB achieves optimal sample efficiency with respect to state and action sizes. The method is further generalized to the multi-agent setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper presents a significant theoretical contribution by establishing optimal sample complexity bounds for robust Nash equilibrium computation in offline RTZMGs. The use of robust unilateral clipped concentrability (Definition 4.1) is novel and tighter than previous assumptions (e.g., in P2M2PO).\n- The method is carefully constructed with a dual formulation for value computation, a tailored penalty function, and proofs of optimality across uncertainty regimes.\n- First optimal bound in robust MARL under offline constraints\n\nWeaknesses\n- While the paper is positioned as theoretical, it includes a small experimental section that aims to support the theoretical results. However, the experiments are limited to synthetic, low-dimensional settings with no real-world applicability or comparison to existing baselines like P2M2PO. As such, the empirical section offers limited additional insight and does not strongly validate the practical robustness of the method.\n- The paper could benefit from a short discussion on how to move toward larger and practical domains.\n\nQUESTIONS:\n1. The concentrability coefficient C_{r}^* plays a central role in your bounds. Is there a principled way to estimate or bound C_{r}^* from offline data in practice, especially when the robust optimal policies are unknown?\n\n2. Solving robust zero-sum matrix games is generally PPAD-hard. In your implementation of RTZ-VI-LCB, how do you compute the robust NE at each state and timestep efficiently? Are there computational relaxations or approximations used?\n\n3. Your theoretical contribution is strong, but the empirical validation is limited to synthetic tabular environments. Do you envision obstacles in extending RTZ-VI-LCB to high-dimensional? What are the main challenges in applying your framework to more realistic offline multi-agent domains?\n\n4. The proposed Bernstein-style penalty and two-stage subsampling are key components to deal with partial coverage and statistical dependence. Could you provide insights into how much each of these components contributes to the algorithm's performance and stability?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses robust two-player zero-sum Markov games (TZMGs) by introducing the RTZ-VI-LCB algorithm, under the offline setting.\n\nSTRENGTHS AND WEAKNESSES:\nS: (1). Robust Markov game is a very interesting topic with growing research interests. \n(2). The paper provides a rigorous theoretical framework, including upper and lower sample complexity bounds. The upper bound is near-optimal. \n\nW: (1). The major weakness is on the limited technical novelty of the work. The algorithm is an direct extension of [a], together with standard LCB technique for offline RL. \n(2). Some references on offline single-agent robust RL are missing, e.g., [b-e]. Also a work [f] on robust games: \n \n[a] Model-Based Reinforcement Learning for Offline Zero-Sum Markov Games\nY Yan, G Li, Y Chen, J Fan - arXiv preprint arXiv:2206.04044, 2022\n\n[b] Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation\nZ Liu, P Xu\nInternational Conference on Artificial Intelligence and Statistics, 2719-2727\n\n[c] Minimax optimal and computationally efficient algorithms for distributionally robust offline reinforcement learning\nZ Liu, P Xu\narXiv preprint arXiv:2403.09621\n\n[d] A unified principle of pessimism for offline reinforcement learning under model mismatch\nY Wang, Z Sun, S Zou\nAdvances in Neural Information Processing Systems 37, 9281-9328\n\n[e]Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity\nL Shi, Y Chi - Journal of Machine Learning Research, 2024\n\n[f]Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning\nL Shi, J Gai, E Mazumdar, Y Chi, A Wierman - arXiv preprint arXiv:2409.20067, 2024\n\nQUESTIONS:\n(1). Can you explain the major difference of your method compared to [a]+offline LCB approach for robust RL?\n(2). Can you connect your result together with the generative model setting? Namely, if the dataset is generated based on a uniform distribution, what will the $C^*_r$ term become, and what is the resulting result? And how is that result compared to [34]?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The authors of this work develop a new lower bound on the sample complexity of robust MARL. They also present an offline algorithm nearly whose sample complexity is near optimal based on the lower bound.

STRENGTHS AND WEAKNESSES:
The brand new lower bound is very significant to the robust MARL literature, and the near-optimal attaining algorithm presents new ideas that push the field forward. The primary downside to this model is the adherence to the (s,a,b)-rectangular model of uncertainty, and the potential computational difficulties that typically arise in such works. Despite these downsides, the results presented are still significant.

QUESTIONS:
It is known that computing robust policies and even approximating them is NP-hard in many regimes. What is the time complexity of your algorithm and do the authors foresee a potential tradeoff between time complexity and sample complexity for robust settings? Also, do the authors believe their results could potentially extend to uncertainty sets beyond (s,a,b) rectangularity? In particular, other works have been able to handle more complex uncertainty sets through a relationship with regularized Markov games.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03