Title: Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributionally robust offline reinforcement learning

double pessimism

general function approximation
Abstract: We study distributionally robust offline reinforcement learning (RL), which seeks to find an optimal robust policy purely from an offline dataset that can perform well in perturbed environments. We propose a generic algorithm framework Doubly Pessimistic Model-based Policy Optimization ($\texttt{P}^2\texttt{MPO}$) for robust offline RL, which features a novel combination of a flexible model estimation subroutine and a doubly pessimistic policy optimization step. Here the double pessimism principle is crucial to overcome the distribution shift incurred by i) the mismatch between behavior policy and the family of target policies; and ii) the perturbation of the nominal model. Under certain accuracy assumptions on the model estimation subroutine, we show that $\texttt{P}^2\texttt{MPO}$ is provably sample-efficient with robust partial coverage data, which means that the offline dataset has good coverage of the distributions induced by the optimal robust policy and perturbed models around the nominal model. By tailoring specific model estimation subroutines for concrete examples including tabular Robust Markov Decision Process (RMDP), factored RMDP, and RMDP with kernel and neural function approximations, we show that $\texttt{P}^2\texttt{MPO}$ enjoys a $\tilde{\mathcal{O}}(n^{-1/2})$ convergence rate, where $n$ is the number of trajectories in the offline dataset. Notably, these models, except for the tabular case, are first identified and proven tractable by this paper. To the best of our knowledge, we first propose a general learning principle --- double pessimism --- for robust offline RL and show that it is provably efficient in the context of general function approximations.

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

Abstract

We study distributionally robust offline reinforcement learning (RL), which seeks to find an optimal robust policy purely from an offline dataset that can perform well in perturbed environments. We propose a generic algorithm framework Doubly Pessimistic Model-based Policy Optimization ( P 2 MPO ) for robust offline RL, which features a novel combination of a flexible model estimation subroutine and a doubly pessimistic policy optimization step. Here the double pessimism principle is crucial to overcome the distribution shift incurred by i) the mismatch between behavior policy and the family of target policies; and ii) the perturbation of the nominal model. Under certain accuracy assumptions on the model estimation subroutine, we show that P 2 MPO is provably sample-efficient with robust partial coverage data , which means that the offline dataset has good coverage of the distributions induced by the optimal robust policy and perturbed models around the nominal model. By tailoring specific model estimation subroutines for concrete examples including tabular Robust Markov Decision Process (RMDP), factored RMDP, and RMDP with kernel and neural function approximations, we show that P 2 MPO enjoys a ˜ O ( n -1 / 2 ) convergence rate, where n is the number of trajectories in the offline dataset. Notably, these models, except for the tabular case, are first identified and proven tractable by this paper. To the best of our knowledge, we first propose a general learning principle - double pessimism - for robust offline RL and show that it is provably efficient in the context of general function approximations.

1 Introduction

Reinforcement learning (RL) [52] aims to learn an optimal policy that maximizes the cumulative rewards received in an unknown environment. Typically, deep RL algorithms learn a policy in an online trial-and-error fashion using millions to billions of data. However, data collection could be costly and risky in some practical applications such as healthcare [56] and autonomous driving [38]. To tackle this challenge, offline RL (also known as batch RL) [21, 22] learns a near-optimal policy based on a dataset collected a priori without further interactions with the environment. Although there has been great progress in offline RL [72, 20, 16, 54, 62, 6], these works implicitly require that the offline dataset is generated by the real-world environment, which may fail in practice. Taking robotics [18, 37] as an example, the experimenter trains agents in a simulated physical environment and then deploy them in real-world environments. Since the experimenter does not have access to the true physical environment, there is a mismatch between the simulated environment used to generate the offline dataset and the real-world environment used to deploy the agents. Such a mismatch is

commonly referred to as the sim-to-real gap [42, 76]. Since the optimal policy is sensitive to the model [31, 9], the potential sim-to-real gap may lead to the poor performance of RL algorithms.

A promising solution to remedy this issue is robust RL [13, 9, 32] - training a robust policy that performs well in a bad or even adversarial environment. A line of work on deep robust RL [43, 44, 41, 30, 53, 75, 19] demonstrates the superiority of the trained robust policy in real world environments. Furthermore, the recent work of Hu et al. [11] theoretically proves that the ideal robust policy can attain near optimality in dealing with problems with sim-to-real gap, but this work does not suggest how to learn a robust policy from a theoretical perspective. In order to understand robust RL from the theoretical side, robust Markov decision process (RMDP) [13, 9] has been proposed, and many recent works [77, 47, 29] design sample-efficient learning algorithms for robust offline RL. These works mainly focus on the tabular case, which is not capable of tackling large state spaces. Meanwhile, in the non-robust setting, a line of works [16, 54, 62, 73, 45] show that 'pessimism' is the general learning principle for designing algorithms that can overcome the distributional shift problem faced by offline RL. In particular, in the context of function approximation, Xie et al. [62] and Uehara and Sun [54] leverage the pessimism principle and propose generic algorithms in the model-free and model-based fashion, respectively. Hence, it is natural to ask the following questions:

Q1: What is the general learning principle for robust offline RL?

Q2: Based on this learning principle, can we design a generic algorithm for robust offline RL in the context of function approximation?

To answer these two questions, we need to tackle the following two intertwined challenges: (i) distributional shift, that is, the mismatch between offline data distribution and the distribution induced by the optimal robust policy. In robust offline RL, the distributional shift has two sources - behavior policy and perturbed model, where the latter is the unique challenge not presented in non-robust RL; and (ii) function approximation. Existing works mainly focus on the tabular case, and it remains elusive how to add reasonable structure conditions to make RMDPs with large state spaces tractable. Despite these challenges, we answer the aforementioned two questions affirmatively.

Contributions. We study robust offline RL in a general framework, which not only includes existing known tractable S × A -rectangular tabular RMDPs, but also subsumes several newly proposed models (e.g., S × A -rectangular factored RMDPs, S × A -rectangular kernel RMDPs, and S × A -rectangular neural RMDPs) as special cases. Under this general framework, we propose a generic model-based algorithm, dubbed as Doubly Pessimistic Model-based Policy Optimization ( P 2 MPO ), which consists of a model estimation subroutine and a policy optimization step based on doubly pessimistic value estimators. We note that the model estimation subroutine can be flexibly chosen according to structural conditions of specific RMDP examples. Meanwhile, the adoption of doubly pessimistic value estimators in the face of model estimation uncertainty and environment uncertainty plays a key role in overcoming the distributional shift problem in robust offline RL.

From the theoretical perspective, we characterize the optimality of P 2 MPO with partial coverage. In particular, we show that the suboptimality gap of P 2 MPO is upper bounded by the model estimation error (see Condition 3.2) and the robust partial coverage coefficient (see Assumption 3.3). For concrete examples of RMDPs, by customizing specific model estimation mechanisms and plugging them into P 2 MPO , we show that P 2 MPO enjoys a n -1 / 2 convergence rate with robust partial coverage data, where n is the number of trajectories in the offline dataset. In summary, we identify a general learning principle double pessimism -for robust offline RL. Based on this principle, we can perform sample-efficient robust offline RL with robust partial coverage data via general function approximation. See Table 1 for a summary of our results and a comparison with mostly related works.

1.1 Related Works

Robust reinforcement learning in robust Markov decision processes. Robust RL is usually modeled as a robust MDP (RMDP) [13, 9], and its planning has been well studied [13, 9, 65, 60, 57]. Recently, robust RL in RMDPs has attracted considerable attention, and a growing body of works studies this problem in the generative model [68, 39, 49, 58, 69, 66, 7], online setting [59, 3, 8], and offline setting [77, 40, 47, 29]. Our work focuses on robust offline RL, and we provide a more in-depth comparison with Zhou et al. [77], Shi and Chi [47], Ma et al. [29] as follows. Under the full coverage condition (a uniformly lower bounded data distribution), Zhou et al. [77] provide the first sample-efficient algorithm for S × A -rectangular tabular RMDPs. After, Shi and Chi [47]

leverage the pessimism principle and design a sample-efficient offline algorithm that only requires robust partial coverage data for S × A -rectangular tabular RMDPs. Ma et al. [29] propose a new d -rectangular RMDP and develop a pessimistic style algorithm that can find a near-optimal robust policy with partial coverage data. In comparison, we provide a generic algorithm that can not only solve the models in Zhou et al. [77], Shi and Chi [47], Ma et al. [29], but can also tackle various newly proposed RMDP models such as S ×A -rectangular factored RMDP, S ×A -rectangular kernel RMDP, and S × A -rectangular neural RMDP. See Table 1 for a summary. Moreover, we propose a new pessimistic type learning principle 'double pessimism' for robust offline RL. Although Shi et al. [48] and Ma et al. [29] adopt the similar algorithmic idea in tabular or linear settings, neither of them have identified a general learning principle for robust offline RL in the regime of large state space.

Non-robust offline RL and pessimism principle. The line of works on offline RL aims to design efficient learning algorithms that find an optimal policy given an offline dataset collected a priori. Prior works [33, 2, 5] typically require a dataset of full coverage, which assumes that the offline data have good coverage of all state-action pairs. In order to avoid such a strong coverage condition on data, the pessimism principle - being conservative in policy or value estimation of those state-action pairs that are not sufficiently covered by data - has been proposed. Based on this principle, a long line of works [see e.g., 16, 54, 62, 63, 45, 73, 71, 64, 48, 24, 26, 74, 28, 46] propose algorithms that can learn the optimal policy only with the partial coverage data . The partial coverage data only requires to cover the state-action pairs visited by the optimal policy. Among these works, our work is mostly related to the work of Uehara and Sun [54], which proposes a generic model-based algorithm for non-robust offline RL. Our algorithm for robust offline RL is also in a model-based fashion, and our study covers some models such as S × A -rectangular kernel and neural RMDPs whose non-robust counterparts are not studied by Uehara and Sun [54]. More importantly, our algorithm is based on a newly proposed double pessimism principle, which is tailored for robust offline RL and is in parallel with the pessimism principle used in non-robust offline RL. Also, we show that the performance of our proposed algorithm depends on the notion of robust partial coverage coefficient , which is also different from the notions of partial coverage coefficient in previous non-robust offline RL works [16, 62, 54].

1.2 Notations

For any set A , we use 2 A to denote the collection of all the subsets of A . For any measurable space X , we use ∆( X ) to denote the collection of probability measures over X . For any integer n , we use [ n ] to denote the set { 1 , · · · , n } . Throughout the paper, we use D ( ·∥· ) to denote a (pseudo-)distance between two probability measures (or densities). In specific, we define the KL-divergence D KL ( p ∥ q ) between two probability densities p and q over X as

and we define the TV-distance D TV ( p ∥ q ) between two probability densities p and q over X as

Given a function class F equipped with some norm ∥·∥ F , we denote by N [] ( ϵ, F , ∥·∥ F ) the ϵ -bracket number of F , and N ( ϵ, F , ∥ · ∥ F ) the ϵ -covering number of F .

2 Preliminaries

2.1 A Unified Framework of Robust Markov Decision Processes

We introduce a unified framework for studying episodic robust Markov decision processes (RMDP), which we denote as a tuple ( S , A , H, P ⋆ , R, P M , Φ ) . The set S is the state space with possibly infinite cardinality, A is the action space with finite cardinality. The integer H is the length of each episode. The set P ⋆ = { P ⋆ h } H h =1 is the collection of transition kernels where each P ⋆ h : S × A ↦→ ∆( S ) , and R = { R h } H h =1 is the collection of reward functions where each R h : S × A ↦→ [0 , 1] . We use ∆( S ) to note the probability simplex on S (i.e. the space of probability measures with support on S ).

We consider a model-based perspective of reinforcement learning, and we denote P = { P ( ·|· , · ) : S × A ↦→ ∆( S ) } as the space of all transition kernels. The space P M ⊆ P of the RMDP is a realizable model space which contains the transition kernel P ⋆ , i.e., P ⋆ h ∈ P M for any step h ∈ [ H ] . Finally, the RMDP is equipped with a mapping Φ : P M ↦→ 2 P that characterizes the robust set of any transition kernel in P M . Formally, for any transition kernel P ∈ P M , we call Φ ( P ) the robust set of P . One can interpret the transition kernel P ⋆ h ∈ P M as the transition kernel of the training environment, while Φ ( P ⋆ h ) contains all the possible transition kernels of the test environment.

Remark 2.1. The mapping Φ is defined on the realizable model space P M , while for generality we allow the image of Φ to be outside of P M . That is, a ˜ P ∈ Φ ( P ) for some P ∈ P M might be in P c M .

Policy and robust value function. Given an RMDP ( S , A , H, P ⋆ , R, P M , Φ ) , we consider using a Markovian policy to make decision. A Markovian policy π is defined as π = { π h } H h =1 with π h : A ↦→ ∆( S ) for each step h ∈ [ H ] . For simplicity, we use policy to refer to a Markovian policy.

Given any policy π , we define the robust value function of π with respect to any set of transition kernels P = { P h } H h =1 ⊆ P M as the following, for each step h ∈ [ H ] ,

Here V π h ( · ; { ˜ P h } H h =1 ) and Q π h ( · ; { ˜ P h } H h =1 ) are the state-value function and the action-value function [52] of policy π in the standard episodic MDP ( S , A , H, { ˜ P h } H h =1 , R ) ,

where the expectation E { ˜ P h } H h =1 ,π [ · ] is taken with respect to the trajectories induced by the transition kernel { ˜ P h } H h =1 and the policy π . Intuitively, the robust value function of a policy π given transition kernel P is defined as the least expected cumulative reward achieved by π when the transition kernel varies in the robust set of P . This is how an RMDP takes the perturbed models into consideration.

S × A -rectangular robust set and robust Bellman equation. Ideally, we would like to consider robust value function that has recursive expressions, just like the Bellman equation satisfied by (2.3) in a standard MDP [52]. To this end, we focus on a generally adopted kind of robust set in our unified framework, which is called the S × A -rectangular robust set [13].

Assumption 2.2 ( S × A -rectangular robust set) . We assume that the mapping Φ induces S × A -rectangular robust sets. Specifically, the mapping Φ satisfies, for ∀ P ∈ P M ,

for some (pseudo-) distance D ( ·∥· ) on ∆( S ) and some ρ ∈ R + . Intuitively, S × A -rectangular requires that Φ ( P ) gives decoupled robust sets for P ( ·| s, a ) across different ( s, a ) -pairs. The (pseudo-)distance D ( ·∥· ) can be chosen as a ϕ -divergence [68, 10] or a p -Wasserstein-distance [35].

Thanks to the S × A -rectangular assumption on the mapping Φ , the robust value functions (2.1) of any policy π then satisfy a recursive expression, which is called robust Bellman equation [13, 36].

Proposition 2.3 (Robust Bellman equation) . Under Assumption 2.2, for any P = { P h } H h =1 where P h ∈ P M and any π = { π h } H h =1 with π h : S ↦→ ∆( A ) , the following robust Bellman equation holds,

To be self-contained, in Appendix B we provide a detailed proof of the robust Bellman equation in our framework under Assumption 2.2. Equation (2.5) actually says that the infimum over all the transition kernels (recall the definition of V π h,P, Φ in (2.1)) can be decomposed into a 'one-step' infimum over the current transition kernel, i.e., inf ˜ P h ∈ Φ ( P h ) , and an infimum over the future transition kernels, i.e., V π h +1 ,P, Φ . Such a property is crucial to the algorithmic design and theoretical analysis for RMDPs.

2.2 Examples of Robust Markov Decision Processes

In this section, we give concrete examples for the general RMDP framework proposed in Section 2.1. Most existing works on RMDPs hinge on the finiteness assumption on the state space, which fails to deal with prohibitively large or even infinite state space. In our framework, RMDPs can be considered in the paradigm of infinite state space, for which we adopt various powerful function approximation tools including kernel and neural functions. Also, we introduce a new type of RMDP named robust factored MDP, which is a robust extension of standard factored MDPs [17].

Remark 2.4. Besides S × A -rectangular-type robust sets (Assumption 2.2), our unified framework of RMDP can also cover other types of robust sets considered in some previous works as special cases, including S -rectangular robust set [61] and d -rectangular robust set for linear MDPs [29]. See Section A for a discussion about these two types of robust sets.

In the sequel, we introduce concrete examples of our framework of RMDP.

Example 2.5 ( S × A -rectangular robust tabular MDP) . When the state space S is a finite set, we call the corresponding model an S × A -rectangular robust tabular MDP. Recently, there is a line of works on the S ×A -rectangular robust tabular MDP [77, 68, 39, 27, 47, 40, 8, 10, 35, 58, 69, 66, 7]. For S × A -rectangular robust tabular MDPs, we choose P M = P containing all possible transitions.

Remark 2.6. We point out that our framework of RMDP under S×A -rectangular assumption covers substantially more model than S × A -rectangular robust tabular MDP since our state space S can be infinite. The model space P M can be adapted to function approximation methods to handle the infinite state space. Thus any efficient algorithm developed for our framework of RMDPs can not be covered by algorithms for S × A -rectangular robust tabular MDPs. Example 2.7 and 2.8 are infinite state space S × A -rectangular robust MDPs with function approximations.

Example 2.7 ( S × A -rectangular robust MDP with kernel function approximations) . We consider an infinite state space S × A -rectangular robust MDP whose realizable model space P M is in a reproduced kernel Hilbert space (RKHS). Let H be a RKHS associated with a positive definite kernel K : ( S × A × S ) × ( S × A × S ) ↦→ R + (See Appendix D.3.1 for a review of the basics of RKHS). We denote the feature mapping of H by ψ : S × A × S ↦→ H . With H , an S × A -rectangular robust MDP with kernel function approximation is defined as an S × A -rectangular robust MDP with

for some B K > 0 . Here we implicitly identify P ( ·|· , · ) as the density of the corresponding distribution with respect to a proper base measure on S × A × S .

Example 2.8 ( S × A -rectangular robust MDP with neural function approximations) . We consider an infinite state space S×A -rectangular robust MDP whose realizable model space P M is parameterized by an overparameterized neural network. We first define a two-layer fully-connected neural network on some X ⊆ R d X as

where m ∈ N + is the number of hidden units of NN , ( W , a ) is the parameters given by W = ( w 1 , · · · , w m ) ∈ R d × m , a = ( a 1 , · · · , a m ) ⊤ ∈ R m , and σ ( · ) is the activation function. Now we assume that the state space S ⊆ R d S for some d S ∈ N + . Also, we identify actions via one-hot vectors in R |A| , i.e., we represent a ∈ A by (0 , · · · , 0 , 1 , 0 , · · · , 0) with 1 in the a -th coordinate. Let X = S ×A×S with d X = 2 d S + |A| . Then an S ×A -rectangular robust MDP with neural function approximation is defined as an S × A -rectangular robust MDP with P M given by

for some B N > 0 and some fixed ( W 0 , a 0 ) which can be interpreted as the initialization. We refer to Appendix D.4.1 for more details about neural function approximations and analysis techniques.

Example 2.9 ( S × A -rectangular robust factored MDP) . We consider a factored MDP equipped with S × A -rectangular factored robust set. A standard factored MDP [17] is defined as follows. Let d be an integer and O be a finite set. The state space S is factored as S = O d . For each i ∈ [ d ] , s [ i ] is the i -coordinate of s and it is only influenced by s [pa i ] , where pa i ⊆ [ d ] . That is, the transition of a factored MDP can be factorized as

Here we let the realizable model space P M consist of all the factored transition kernels, i.e.,

For an S×A -rectangular robust factored MDP, we define Φ as, for any transition kernel P ( s ′ | s, a ) = ∏ d i =1 P i ( s ′ [ i ] | s [pa i ] , a ) ∈ P M , Φ ( P ) = ⊗ ( s,a ) ∈S×A P Fac ,ρ ( s, a ; P ) , with

for some (pseudo-)distance D on ∆( O ) and some positive real numbers { ρ i } d i =1 .

2.3 Offline Reinforcement Learning in Robust Markov Decision Processes

In this section, we define the offline RL protocol in a RMDP ( S , A , H, P ⋆ , R, P M , Φ ) . The learner is given the realizable model space P M and the robust mapping Φ , but the learner doesn't know the transition kernel P ⋆ . For simplicity, we assume that the learner knows the reward function R 2 .

Offline dataset generation. We assume that the learner is given an offline dataset D that consists of n i.i.d. trajectories generated from the standard MDP ( S , A , H, P ⋆ , R ) using some behavior policy π b . For each τ ∈ [ n ] , the trajectory has the form of { ( s τ h , a τ h , r τ h ) } H h =1 , satisfying that a τ h ∼ π b h ( ·| s τ h ) , r τ h = R h ( s τ h , a τ h ) , and s τ h +1 ∼ P ⋆ h ( ·| s τ h , a τ h ) for each step h ∈ [ H ] .

Given transition kernels P = { P h } H h =1 and a policy π , we use d π P,h ( · , · ) to denote the state-action visitation distribution at step h when following policy π and transition kernel P . With this notation, the distribution of ( s τ h , a τ h ) can be written as d π b P ⋆ ,h or simply d b P ⋆ ,h , for each τ ∈ [ n ] and h ∈ [ H ] . We also use d π b P ⋆ ,h ( · ) to denote the marginal distribution of state at step h when there is no confusion.

Learning objective. In offline robust RL, the goal is to learn the policy π ⋆ from the offline dataset D which maximizes the robust value function V π 1 ,P ⋆ , Φ , that is,

where Π = { π = { π h } H h =1 | π h : S ↦→ ∆( A ) } denotes the collection of all Markovian policies. In view of (2.10), we call π ⋆ the optimal robust policy . Equivalently, we want to learn a policy ̂ π ∈ Π which minimizes the suboptimality gap between ̂ π and π ⋆ , defined as 3

Algorithm 1 Doubly Pessimistic Model-based Policy Optimization ( P 2 MPO )

1: Input : model space P M , mapping Φ , dataset D , policy class Π , algorithm ModelEst .

2: Model estimation step:

3: Obtain a confidence region ̂ P = ModelEst ( D , P M ) .

4: Doubly pessimistic policy optimization step:

5: Set policy ̂ π as argsup π ∈ Π J Pess 2 ( π ) , where J Pess 2 ( π ) is defined in (3.1).

6: Output : ̂ π = { ̂ π h } H h =1 .

3 Algorithm: Generic Framework and Unified Theory

In this section, we propose Doubly Pessimistic Model-based Policy Optimization ( P 2 MPO ) algorithm to solve offline RL in the RMDP framework we introduce in Section 2.1, and we establish a unified theoretical guarantee for P 2 MPO . Our proposed algorithm and theory show that double pessimism is a general principle for designing efficient algorithms for offline robust RL. The algorithm features three key points: i) learning the optimal robust policy π ⋆ approximately; ii) requiring only a partial coverage property of the offline dataset D ; iii) able to handle infinite state space via function approximations.

We first introduce our proposed algorithm framework P 2 MPO in Section 3.1. Then we establish a unified analysis for P 2 MPO in Section 3.2. Our algorithm framework can be specified to solve all the concrete examples of RMDP we introduce in Section 2.2, which we show in Section 4.

3.1 Algorithm Framework: P 2 MPO

The P 2 MPO algorithm framework (Algorithm 1) consists of a model estimation step and a doubly pessimistic policy optimization step , which we introduce in the following respectively.

Model estimation step (Line 3). The P 2 MPO algorithm framework first constructs an estimation of the transition kernels P ⋆ = { P ⋆ h } H h =1 , i.e., it estimates the dynamic of the training environment. It implements a sub-algorithm ModelEst ( D , P M ) that returns a confidence region ̂ P for P ⋆ = { P ⋆ h } H h =1 . Specifically, ̂ P = { ̂ P h } H h =1 with ̂ P h ⊆ P M for each step h ∈ [ H ] .

The sub-algorithm ModelEst can be tailored to specific RMDPs. We refer to Section 4 for detailed implementations of ModelEst for different examples of RMDPs introduced in Section 2.2. Ideally, to ensure sample-efficient learning, we need ̂ P = ModelEst ( D , P M ) to satisfy: i) the transition kernels P ⋆ = { P ⋆ h } H h =1 are contained in ̂ P = { ̂ P h } H h =1 ; ii) each transition kernel P h ∈ ̂ P h enjoys a small 'robust estimation error' which is highly related to the robust Bellman equation in (2.5). We quantify these two conditions of ̂ P for sample-efficient learning in Section 3.2.

Doubly pessimistic policy optimization step (Line 5). After model estimation step , P 2 MPO performs policy optimization to find the optimal robust policy. To learn the optimal robust policy in the face of uncertainty, P 2 MPO adopts a double pessimism principle. To explain, this general principle has two sources of pessimism: i) pessimism in the face of data uncertainty; ii) pessimism to find a robust policy. Specifically, for any policy, we first estimate its robust value function via two infimums, where one is an infimum over the confidence set constructed in the model estimation step, and one is an infimum over the robust sets. Formally, for any π ∈ Π , we define the doubly pessimistic estimator

where V π 1 is the standard value function of policy π defined in (2.3). Then P 2 MPO outputs the policy ̂ π that maximizes the doubly pessimistic estimator J Pess 2 ( π ) defined in (3.1), i.e.,

The novelty of the doubly pessimistic policy optimization step is performing pessimism from the two sources (data uncertainty and robust optimization) simultaneously. Compared with the previous works on standard offline RL in MDPs [62, 54] and offline RL in RMDPs without pessimism [68, 77, 40], they only contain one source of pessimism in algorithm design, contrasting with our algorithm.

We note that a recent work [47] also studied robust offline RL in S × A -rectangular robust tabular MDPs (Example 2.5) using pessimism techniques. Compared with our double pessimism principle,

their algorithm performs pessimism in face of data uncertainty i) depending on the tabular structure of the model since a point-wise pessimism penalty is needed and ii) depending on the specific form of the robust set Φ ( P ) , which makes it difficult to adapt to the infinite state space case with general function approximations and general types of robust set Φ ( P ) .

3.2 Unified Theoretical Analysis

In this section, we establish a unified theoretical analysis for the P 2 MPO algorithm framework proposed in Section 3.1. We first specify the two conditions that the model estimation step of P 2 MPO should satisfy in order for sample-efficient learning. Then we establish an upper bound of suboptimality of the policy obtained by P 2 MPO given that these two conditions are satisfied. In Section 4, we show that the specific implementations of the sub-algorithm ModelEst for the RMDPs examples in Section 2.2 satisfy these two conditions, which results in tailored suboptimality bounds for these examples.

Conditions. The two conditions on the model estimation step are given by the following.

Condition 3.1 ( δ -accuracy). With probability at least 1 -δ , it holds that P ⋆ h ∈ ̂ P h for any h ∈ [ H ] .

Condition 3.2 ( δ -model estimation error). With probability at least 1 -δ , it holds that

Condition 3.1 requires that the confidence region ̂ P h contains the transition kernel of the training environment P ⋆ h with high probability. Condition 3.2 requires that each transition kernel P h ∈ ̂ P h induces an error from P ⋆ h less than certain quantity Err Φ h ( n, δ ) , where the error is adapted from the robust Bellman equation (2.5) and involves an infimum over the robust set of P h and P ⋆ h . In specific implementations of ModelEst for RMDP examples in Section 4, we show that the quantity Err Φ h ( n, δ ) generally scales with ˜ O ( n -1 ) , where n is the number of trajectories in the offline dataset.

Suboptimality analysis. Now we establish a unified suboptimality bound for the P 2 MPO algorithm framework. Thanks to the double pessimism principle of P 2 MPO , we can prove a suboptimality bound while only making a mild robust partial coverage assumption on the dataset.

Assumption 3.3 (Robust partial coverage) . We assume that

and we call C ⋆ P ⋆ , Φ the robust partial coverage coefficient.

To interpret, Assumption 3.3 only requires that the dataset covers the visitation distribution of the optimal policy π ⋆ , but in a robust fashion since C ⋆ P ⋆ , Φ considers all possible transition kernels in the robust set Φ ( P ⋆ ) . The robust consideration in C ⋆ P ⋆ , Φ is because in RMDPs the policies are all evaluated in a robust way. This partial-coverage-style assumption is much weaker than full-coveragestyle assumptions [68, 77, 40] which require either a uniformly lower bounded dataset distribution or covering the visitation distribution of any π ∈ Π . For S × A -rectangular robust tabular MDPs (Example 2.5), the robust partial coverage coefficient C ⋆ P ⋆ , Φ is similar with the partial coverage coefficient proposed by [47] who studied tabular RMDPs under partial coverage. We highlight that beyond S × A -rectangular robust tabular MDPs, our robust partial coverage assumption can handle other examples of RMDPs (Section 2.2) under our unified theory.

Our main result is the following theorem. See Appendix C for a detailed proof.

Theorem 3.4 (Suboptimality of P 2 MPO ) . Under Assumptions 2.2 and 3.3, suppose that Algorithm 1 implements a sub-algorithm that satisfies Conditions 3.1 and 3.2, then with probability at least 1 -2 δ ,

When Err Φ h ( n, δ ) achieves a rate of ˜ O ( n -1 ) , then P 2 MPO enjoys a ˜ O ( n -1 / 2 ) -suboptimality. In the following Section 4, we give specific implementations of the model estimation step of P 2 MPO for each example of RMDP in Section 2. The implementations will make Conditions 3.1 and 3.2 satisfied and thus specify the unified result Theorem 3.4.

4 Implementations of P 2 MPOfor Examples of RMDPs

In this section, we provide concrete implementations of the ModelEst sub-algorithm in P 2 MPO (Algorithm 1). In Section 4.1, we implement ModelEst for all the RMDPs that satisfy Assumption 2.2, and we specify the suboptimality bounds in Theorem 3.4 to Examples 2.5, 2.7, 2.8 in Section 2.2. In Section 4.2, we implement ModelEst for S × A -rectangular robust factored MDPs (Example 2.9) and specify Theorem 3.4 to this example.

4.1 Model Estimation for General RMDPs with S × A -rectangular Robust Sets

Using the offline data D , we first construct the maximum likelihood estimator (MLE) of the transition kernel P ⋆ . Specifically, for each step h ∈ [ H ] , we define

After, we construct a confidence region for the MLE estimator, denoted by ̂ P . Specifically, ̂ P contains all transitions which have a small total variance distance from ̂ P . For each step h ∈ [ H ] , we define

Here ξ > 0 is a tuning parameter that controls the size of the confidence region ̂ P h . Finally, we set ModelEst ( D , P M ) = ̂ P = { ̂ P h } H h =1 with ̂ P h given in (4.2). In the sequel, we mainly consider the distance D ( ·∥· ) in Assumption 2.2 to be KL-divergence and TV-distance. The following corollary specifies Theorem 3.4 to model estimation step given by (4.2). See Appendix D for a detailed proof. Corollary 4.1 (Suboptimality of P 2 MPO : S × A -rectangular robust MDP) . Under Assumption 2.2, 3.3, setting the tuning parameter ξ as

for some constants C 1 , C 2 > 0 , P 2 MPO with model estimation step given by (4.2) satisfies that

♠ when D ( ·∥· ) is KL-divergence and Assumption D.3 holds with parameter λ , then with probability at least 1 -2 δ ,

♠ when D ( ·∥· ) is TV-divergence, then with probability at least 1 -2 δ ,

S × A -rectangular robust tabular MDP (Example 2.5). When S is finite as in Example 2.5, the MLE estimator (4.1) coincides the empirical estimator

which is adopted by [77, 68, 39, 47, 40]. Furthermore, in Example 2.5, the realizable model space P M = { P : S × A ↦→ ∆( S ) } . When S and A are finite, we can bound the bracket number of P M as

Combining (4.4) and Corollary 4.1, we can conclude that: i) under TV-distance the suboptimality of P 2 MPO for S×A -rectangular robust tabular RMDP is given by O ( H 2 √ C ⋆ P ⋆ , Φ |S| 2 |A| log( nH/δ ) /n ) , ii) under KL-divergence the suboptimality of P 2 MPO for S × A -rectangular robust tabular MDP is given by O ( H 2 exp( H/λ ) /ρ · √ C ⋆ P ⋆ , Φ |S| 2 |A| log( nH/δ ) /n ) . We prove (4.4) in Appendix D.2.

Remark 4.2. We note that for KL-divergence robust sets, the dependence on exp( H ) is due to the usage of general function approximations, which also appears in a recent work [29] for RMDPs with linear function approximations. For the special case of robust tabular MDPs, under KLdivergence, existing work [47] derived sample complexities without exp( H ) , but with an additional dependence on 1 /d b min and 1 /P ⋆ min Here d b min = min ( s,a,h ): d π b ( s,a ) > 0 d π b P ⋆ ,h ( s, a ) and P ⋆ min =

P ⋆ ,h min ( s,s ′ ,h ): P h ( s ′ | s,π ⋆ h ( s )) > 0 P ⋆ h ( s ′ | s, π ⋆ h ( s )) . We remark that our analysis for P 2 MPO algorithm can be tailored to the tabular case and become exp( H ) -free using their techniques, with the cost of an additional dependence on 1 /d b min and 1 /P ⋆ min . But we note that in the infinite state space case, both the 1 /d b min -dependence and the 1 /P min -dependence becomes problematic. So, it serves as an interesting future work to answer whether one can derive both exp( H ) -free and (1 /d b min , 1 /P ⋆ min ) -free results for (general) function approximations under KL-divergence.

S × A -rectangular robust MDP with kernel and neural function approximations (Examples 2.7 and 2.8). By specifying the bracket numbers in Corollary 4.1, we can provide the detailed suboptimality guarantees for S ×A -rectangular robust MDP with kernel and neural function approximations. Due to space limitations, we defer the detailed results to Appendices D.3 and D.4.

4.2 Model Estimation for S × A -rectangular Robust Factored MDPs (Example 2.9)

We first construct MLE estimator for each factor P ⋆ h,i of the transition P ⋆ h = ∏ d i =1 P ⋆ h,i , that is,

Then given { ̂ P h,i } d i =1 we construct a confidence region that is factored across i ∈ [ d ] . Specifically,

Finally, we set ModelEst ( D , P M ) = ̂ P = { ̂ P} H h =1 with ̂ P h given in (4.6). The following corollary specifies Theorem 3.4 to model estimation step given by (4.6). See Appendix E for a detailed proof. Corollary 4.3 (Suboptimality of P 2 MPO : S × A -rectangular robust factored MDP) . Supposing the RMDP is an S × A -rectangular robust factored MDP, under the same Assumptions and parameter choice in Theorem 3.4 and Proposition E.1, P 2 MPO with model estimation step given by (4.6) satisfies

♣ when D ( ·∥· ) is KL-divergence and Assumption D.3 holds with parameter λ , then with probability at least 1 -2 δ , (defining ρ min = min i ∈ [ d ] ρ i )

♣ when D ( ·∥· ) is TV-divergence, then with probability at least 1 -2 δ ,

Compared with the suboptimality bounds for S × A -rectangular robust MDPs in Section 4.1, the suboptimality of S × A -rectangular robust factored MDPs with ModelEst given in (4.6) only scales with ∑ d i =1 |O| 1+ | pa i | instead of scaling with |S| = ∏ d i =1 |O| which is of order exp( d ) . This justifies the benefit of considering S × A -rectangular robust factored MDPs when the transition kernels of training and testing environments enjoy factored structures.

5 Conclusion and Discussions

This paper proposes a general learning principle - double pessimism - for robust offline RL. Based on this learning principle, we propose a generic algorithm that only requires robust partial coverage data to solve S × A -rectangular RMDPs with general function approximation. Our results are ready to be extended to d -rectangular linear RMDPs [29]. See Appendix A for details. In Appendix A, we also provide some challenges to perform sample efficient RL in S -rectangular RMDPs.

Acknowledgments and Disclosure of Funding

The material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397. Additional support is gratefully acknowledged from NSF 1915967, 2118199, 2229012, 2312204.

References

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Andr´ as Antos, Csaba Szepesv´ ari, and R´ emi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning , 71(1):89-129, 2008.

Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520. PMLR, 2021.

Qi Cai, Zhuoran Yang, Csaba Szepesvari, and Zhaoran Wang. Optimistic policy optimization with general function approximations. 2020.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.02446 , 2022.

Pierre Clavier, Erwan Le Pennec, and Matthieu Geist. Towards minimax optimality of modelbased robust reinforcement learning. arXiv preprint arXiv:2302.05372 , 2023.

Jing Dong, Jingwei Li, Baoxiang Wang, and Jingzhao Zhang. Online policy optimization for robust mdp. arXiv preprint arXiv:2209.13841 , 2022.

Laurent El Ghaoui and Arnab Nilim. Robust solutions to markov decision problems with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Robust ϕ -divergence mdps. 2022.

Jiachen Hu, Han Zhong, Chi Jin, and Liwei Wang. Provable sim-to-real transfer in continuous domain with partial observations. arXiv preprint arXiv:2210.15598 , 2022.

Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online , pages 1695-1724, 2013.

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2): 257-280, 2005.

Arthur Jacot, Franck Gabriel, and Cl´ ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI , volume 16, pages 740-747, 1999.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7247-7254, 2022.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33: 1179-1191, 2020.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning , pages 45-73. Springer, 2012.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Chris Junchi Li, Dongruo Zhou, Quanquan Gu, and Michael I Jordan. Learning two-player mixture markov games: Kernel function approximation and correlated equilibrium. arXiv preprint arXiv:2208.05363 , 2022.

Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 , 2022.

Zhihan Liu, Miao Lu, Zhaoran Wang, Michael Jordan, and Zhuoran Yang. Welfare maximization in competitive equilibrium: Reinforcement learning for markov exchange economy. In International Conference on Machine Learning , pages 13870-13911. PMLR, 2022.

Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Learning from demonstration: Provably efficient adversarial policy imitation with linear function approximation. In International Conference on Machine Learning , pages 14094-14138. PMLR, 2022.

Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust q -learning. In International Conference on Machine Learning , pages 13623-13643. PMLR, 2022.

Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang. Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. arXiv preprint arXiv:2205.13589 , 2022.

Xiaoteng Ma, Zhipeng Liang, Li Xia, Jiheng Zhang, Jose Blanchet, Mingwen Liu, Qianchuan Zhao, and Zhengyuan Zhou. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2209.06620 , 2022.

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausible perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3932-3939. IEEE, 2017.

Shie Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis. Bias and variance in value function estimation. In Proceedings of the twenty-first international conference on Machine learning , page 72, 2004.

Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation , 17(2): 335-359, 2005.

R´ emi Munos. Error bounds for approximate value iteration. In Proceedings of the National Conference on Artificial Intelligence , volume 20, page 1006. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.

Gergely Neu and Ciara Pike-Burke. A unifying view of optimism in episodic reinforcement learning. Advances in Neural Information Processing Systems , 33:1392-1403, 2020.

Ariel Neufeld and Julian Sester. Robust q -learning algorithm for markov decision processes under wasserstein uncertainty. arXiv preprint arXiv:2210.00898 , 2022.

Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafał J´ ozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. CoRR , 2018. URL http://arxiv.org/abs/1808.00177 .

Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. arXiv preprint arXiv:1709.07174 , 2017.

Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics , pages 9582-9602. PMLR, 2022.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. arXiv preprint arXiv:2208.05129 , 2022.

Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632 , 2017.

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018.

Lerrel Pinto, James Davidson, and Abhinav Gupta. Supervision via competition: Robot adversaries for learning tasks. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 1601-1608. IEEE, 2017.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning , pages 2817-2826. PMLR, 2017.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716 , 2022.

Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 , 2022.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint arXiv:2202.13890 , 2022.

Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust batch contextual bandits. Management Science , 2023.

Ingo Steinwart and Andreas Christmann. Support vector machines . Springer Science & Business Media, 2008.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference on learning theory , pages 2898-2933. PMLR, 2019.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR, 2019.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 , 2021.

Sara A Van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2447-2456, 2018.

Qiuhao Wang, Chin Pang Ho, and Marek Petrik. On the convergence of policy gradient in robust mdps. arXiv preprint arXiv:2212.10439 , 2022.

Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. A finite sample complexity bound for distributionally robust q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3370-3398. PMLR, 2023.

Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems , 34:7193-7206, 2021.

Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning , pages 23484-23526. PMLR, 2022.

Wolfram Wiesemann, Daniel Kuhn, and Berc ¸ Rustem. Robust markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellmanconsistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, and Tong Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Singleagent mdp and markov game. arXiv preprint arXiv:2205.15512 , 2022.

Huan Xu and Shie Mannor. Distributionally robust markov decision processes. Advances in Neural Information Processing Systems , 23, 2010.

Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 9728-9754. PMLR, 2023.

Greg Yang and Hadi Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907.10599 , 2019.

Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Towards theoretical understandings of robust markov decision processes: Sample complexity and asymptotics. arXiv preprint arXiv:2105.03863 , 2021.

Wenhao Yang, Han Wang, Tadashi Kozuno, Scott M Jordan, and Zhihua Zhang. Avoiding model estimation in robust markov decision processes with a generative model. arXiv preprint arXiv:2302.01248 , 2023.

Zhuoran Yang, Chi Jin, Zhaoran Wang, Mengdi Wang, and Michael Jordan. Provably efficient reinforcement learning with kernel and neural function approximations. Advances in Neural Information Processing Systems , 33:13903-13916, 2020.

Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34:4065-4078, 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34:13626-13640, 2021.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and ChoJui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037, 2020.

Wenshuai Zhao, Jorge Pe˜ na Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI) , pages 737-744. IEEE, 2020.

Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3331-3339. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a unified framework of Robust Markov Decision Process, which includes many newly proposed models as special cases. Under their generic framework, they proposed Doubly Pessimistic Model-based Policy Optimization (P^2MPO) that adopts a double pessimism principle for policy optimization. They showed that the suboptimality gap of the proposed algorithm can be upper bounded by the model estimation error and the robust partial coverage coefficient. They further provided concrete implementations of their algorithm on some specific models and showed that their algorithm enjoys a $n^{-1/2}$ convergence rate with $n$ being the number of trajectories in the offline dataset.\n\nSTRENGTHS:\n- They proposed a generic framework that covers many models such as $\\mathcal{S}\\times\\mathcal{A}$-rectangular tabular RMDPs, $\\mathcal{S}\\times\\mathcal{A}$-rectangular kernel RMDPs, $\\mathcal{S}\\times\\mathcal{A}$-rectangular neural RMDPs, $\\mathcal{S}\\times\\mathcal{A}$-rectangular factored RMDPs.\n- Instead of covering the visitation distribution of any policy, they only require a partial coverage style assumption, i.e., the dataset covers the visitation distribution of the optimal policy in a robust fashion.\n\nWEAKNESSES:\n- Computationally inefficient, i.e., not sure how to solve (4.1), (4.2) and (3.2) efficiently.\n- Though the authors showed how the model estimation step is implemented on some specific models, it is not clear how to implement the model estimation step that satisfies Condition 3.1 and Condition 3.2 in general.\n\nQUESTIONS:\n- Why the robust partial coverage assumption (which considers all the possible transition kernels in a robust set) is much weaker than assuming the dataset covers the visitation distribution of any possible policy (under only the true transition kernel)? In other words, does the full-coverage-style assumption implies the partial-coverage-style assumption?\n- How does the rates obtained in Section 4 Corollary 4.1, 4.3 compare with the previous works?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study focuses on distributionally robust offline reinforcement learning to discover an optimal robust policy using an offline dataset for effective performance in perturbed environments. A novel algorithm framework called Doubly Pessimistic Model-based Policy Optimization (P2MPO) is proposed. P2MPO combines a flexible model estimation subroutine with a doubly pessimistic policy optimization step, leveraging the double pessimism principle to address challenges arising from behavior-policy mismatch and model perturbations. The study demonstrates the sample efficiency of P2MPO with robust partial coverage data and highlights its convergence rate. It introduces the concept of double pessimism as a general learning principle for robust offline RL, showcasing its efficiency in kernel and neural function approximators.\n\nSTRENGTHS:\nI think this work really pushes the robust RL community research efforts further by answering: \n> can we design a generic algorithm for robust offline RL in the context of function approximation?\n\nThe main contribution of double pessimism: one for robustness of dynamics uncertainty and one for model-estimation using offline data is a really nice idea worthy for publication at NeurIPS.\n\nWEAKNESSES:\nI have only minor weaknesses for this work as follows:\n\n1. The robust Bellman equation for d-rectangular sets are not formally proven. The proof isn't there even in [Ma et al. 22] to the extent of my search. The current Appendix C just rewrites [Iyengar 05] proof. Maybe replace it by the proof for d-rectangular robust Bellman equation?\n2. The robust partial concentrability dependence on the robust set $\\Phi$: Is it tight? Is there a lower bound for this setting? The reason for this question is coming from the fact that the robust solution is looking for $\\min$ over the robust set $\\Phi$.\n3. Honestly, Section B.2 can be expanded further. I found the following statement vague in its current form:\n> our algorithm framework is unable to deal with this kind of rectangular robust sets in the context of partial coverage data due to some technical problems in applying the partial coverage coefficient (Assumption 3.3) under this kind of robust sets.\n\nQUESTIONS:\nplease see weaknesses\n\nLIMITATIONS:\nRegarding S-rectangular sets, please see weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper studies distributionally robust offline reinforcement learning. They propose a general learning principle, double pessimism, as well as a generic algorithm framework P$^2$MPO for robust offline RL, and  show that it is provably efficient in the context of general function approximation.

STRENGTHS:
The paper is well-written and theoretically solid. It provides several novel contributions to the field of distributionally robust MDP.

First, it proposes a general learning principle, double pessimism, together with a generic algorithm framework P$^2$MPO and a unified theoretical analysis.

Second, it proposes several novel structures of uncertainty set and discusses their qualities under three commonly used rectangularity assumptions. Under these structures, they solve the open problem of learning robust offline RL in the context of general function approximation.

WEAKNESSES:
Some minor issue of typos.

QUESTIONS:
1. On line 100, is the policy a function from $\mathcal{S}$ to $\Delta(A)$?

2. How strong are assumption D.1 and E.1, are they reasonable?

3. Check inequality (E.23) '$0\leq \lambda_iH$'.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a generic framework to study distributionally robust offline reinforcement learning problems, which included a model estimation step and a robust policy optimization step. Previous works in the literature usually assume finite state action spaces; this framework can incorporate function approximations and ultimately paved the way of tackling large state action spaces. Additionally, two specific model estimation approaches are introduced and studied in details; corresponding sample complexity results are provided.\n\nSTRENGTHS:\nThis paper explicitly considered the two sources of uncertainties in distirbutionally robust RL problems. The first one is the difference between the estimated training model using finite samples and the true training model; the second one is the difference between the true training model and the testing model. Current literature usually assume that the testing model is within a radius of the estimated training model. The double pessimistic idea is new. Additionally, only partial coverage offline dataset is required, while many previous works require observing all state action pairs in the dataset.  \n\nThis paper also provided an interesting sub-optimality theorem that decomposes the sub-optimality gap into a model estimation error part and a data coverage part, which could be used for future developments. \n\nThe problem setting, definitions and notations are clear and easy to understand.\n\nWEAKNESSES:\nThe paper is somewhat hard to follow. A lot of examples that can be studied under the proposed framework are presented in the main paper. I think it is better to present one example in details in the main paper, move the others to the appendix and put relevant literature review in the main paper. Similarly, in the model estimation parts, one example should be enough. \n\nThe motivation of considering the double pessimism is not clear to me. Two uncertainty sets are introduced, the model uncertainty set, for example, is controlled by $\\epsilon$ when using MLE estimator and the distribution shift robust uncertainty set is controlled by $\\rho$. It is not clear to me whether the two phase uncertainty sets are necessary as in general we don't know $\\rho$. In practice, people may use similar datasets that include both training and testing sets to estimate (guess) the uncertainty set radius, which is a single phase approach and directly captures the difference between the finite training set and the testing set. More rationales or examples should be provided to motivate the two-step approach. For example, 1) the authors could provide applications that singles-step uncertainty set cannot be easily estimated while the two-step approach is more applicable in practice, 2) the author could theoretically show that this two-step approach can avoid conservativeness under certain conditions.\n\nQUESTIONS:\n1.  The underscore $\\lambda$ and $C_1$ in the Corollary 4.1 is not defined in the main paper. They are well-defined in the appendix; I think it is better to at least provided the meaning of underscore $\\lambda$ here. In addition, is it possible to avoid the assumption of the lower bound on the dual variable $\\lambda$? As you consider the two-step uncertainty set, I think you could provide sub-optimality gap in terms of $\\epsilon$ and $\\rho$ in Section 4 and discuss the choices of them as well. \n\n2. Some distributionally robust RL papers that adopt KL-divengence as the uncertainty set measure have radius square in the SubOpt term, e.g., [37] [64] in the papers you cited, while your SubOpt does not suffer from the radius square term. Can you comment on this? I think this could be a big benefit when selecting small $\\rho$s.\n\nLIMITATIONS:\nThis is a theoretical work at this stage and thus no potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper studies the offline robust RL problem. A double pessimism approach is proposed and studied.

STRENGTHS:
1. The approach is novel and new compared to previous offline robust RL ones. 
2. The approach can be used for large-scale problems. 
3. The theoretical analysis is comprehensive.

WEAKNESSES:
1. The model the authors proposed, seems hard to solve. With no robustness consider, the model reduces to the one in [Masatoshi Uehara and Wen Sun, 2022]. The model in the non-robust is hard to solve and becomes hard together with robustness. 
2. Compared to [Masatoshi Uehara and Wen Sun, 2022], the contribution seems a little bit incremental to me. The results and approaches are not surprising, in aspects of approach designing and error bound analysis.

QUESTIONS:
1. Do you have some efficient approach to solve the model you proposed? 
2. The analysis seems similar to the ones in [Masatoshi Uehara and Wen Sun, 2022], can you highlight the novelty and contribution?

LIMITATIONS:
See parts above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04