Title: A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

offline

robust
Abstract: In this paper, we address the challenges of offline reinforcement learning (RL) under model mismatch, where the agent aims to optimize its performance through an offline dataset that may not accurately represent the deployment environment. We identify two primary challenges under the setting: inaccurate model estimation due to limited data and performance degradation caused by the model mismatch between the dataset-collecting environment and the target deployment one. To tackle these issues, we propose a unified principle of pessimism using distributionally robust Markov decision processes. We carefully construct a robust MDP with a single uncertainty set to tackle both data sparsity and model mismatch, and demonstrate that the optimal robust policy enjoys a near-optimal sub-optimality gap under the target environment across three widely used uncertainty models: total variation, $\chi^2$ divergence, and KL divergence. Our results improve upon or match the state-of-the-art performance under the total variation and KL divergence models, and provide the first result for the $\chi^2$ divergence model.

A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch

Abstract

In this paper, we address the challenges of offline reinforcement learning (RL) under model mismatch, where the agent aims to optimize its performance through an offline dataset that may not accurately represent the deployment environment. We identify two primary challenges under the setting: inaccurate model estimation due to limited data and performance degradation caused by the model mismatch between the dataset-collecting environment and the target deployment one. To tackle these issues, we propose a unified principle of pessimism using distributionally robust Markov decision processes. We carefully construct a robust MDP with a single uncertainty set to tackle both data sparsity and model mismatch, and demonstrate that the optimal robust policy enjoys a near-optimal sub-optimality gap under the target environment across three widely used uncertainty models: total variation, χ 2 divergence, and KL divergence. Our results improve upon or match the state-of-the-art performance under the total variation and KL divergence models, and provide the first result for the χ 2 divergence model.

1 Introduction

Reinforcement learning (RL) [40] learns a policy to maximize cumulative rewards through online interactions with the environment. However, in real-world applications such as autonomous vehicles [16] and health care [61], the trial-and-error nature of interacting with the environment can be both costly and dangerous, rendering online learning impractical. To circumvent these challenges, the concept of offline RL has emerged [17, 18], seeking to learn an optimal policy from a pre-collected dataset, eliminating the need for real-time interaction with the environment.

Despite its potential, offline RL faces two significant challenges that impact its performance. The first challenge stems from the nature of the offline dataset itself. Offline RL demonstrates impressive performance only when the dataset is of high quality, as exemplified by [41, 9, 52, 18]. However, in most RL scenarios, the data collection process can be expensive, constrained, and subject to specific behavior policies. This often results in a dataset with insufficient samples and limited coverage. The limited coverage of state-action pairs in the dataset presents a substantial hurdle, making it challenging to fully learn the environment model and the model learned may be inaccurate, leading to a poorly performing policy. and hindering the discovery of the optimal policy, particularly when

Zhongchang Sun

the behavior policy is sub-optimal. Existing solutions involve introducing pessimism during policy learning, by penalizing the reward function for state-action pairs that are not adequately covered by the dataset, known as the lower confidence bound (LCB) approach [15, 33, 19], demonstrating potential both numerically and theoretically in solving offline RL.

The second critical challenge in offline RL pertains to its vulnerability to model mismatch. As the offline dataset is collected in advance, it only encapsulates the environment's information at the time of data collection. In practical applications, environments often undergo variations due to e.g., unexpected perturbations, heterogeneity, or non-stationarity. This inherent variability introduces a model mismatch between the target environment and the one where the dataset is collected, leading to significant performance degradation when deploying the learned policy in the real environment. Robust RL or distributionally robust optimization (DRO) framework is then proposed to address this challenge [13, 28], which incorporates the pessimism principle to effectively tackle model uncertainty. By constructing an uncertainty set containing 'plausible' environments, robust RL optimizes the worst-case performance among them, providing a performance guarantee for the real environment.

To summarize, there are two sources of uncertainty in offline RL: Uncertainty from limited data: This stems from the inherent uncertainty introduced by limited offline datasets and lack of exploration; Uncertainty from model-mismatch: This arises from distribution shifts between data-collection and deployment environments, as well as between data-collection distribution and the distribution induced by the optimal policy. Each challenge can be addressed through its corresponding principle of pessimism, as in previous works: limited data coverage can be mitigated with reward estimation penalties (LCB), while model mismatch can be tackled with distribution estimation penalties (DRO). However, existing approaches often address these uncertainties separately, e.g., [36, 5, 24], leading to methodological redundancy or complexity (more discussions and comparisons can be found in Section 5). In this paper, we propose a unified framework that integrates both principles of pessimism into a single robust Markovian decision process (MDP) model for offline RL, offering a clear and streamlined conceptual formulation and providing improved or matched theoretical guarantees compared to existing methods. Our major contributions can be summarized as follows.

A unified distributionally robust formulation for offline RL under model mismatch. As discussed before, offline RL faces challenges including limited dataset coverage and model mismatch. Specifically, we first tackle the challenge of model mismatch using the approach of distributionally robust MDP, which optimizes the worst-case performance over an uncertainty set specified by e.g., total variation, χ 2 or Kullback-Leibler divergence. We then show that the uncertainty from data sparsity can be transformed into a data-dependent penalization term that is added to the radius of the constructed uncertainty set. Our formulation hence unifies the two principles of pessimism to a single DRO problem that tackles both sources of uncertainty without additional structures and enjoys an easier implementation compared to existing works.

Augmented design of radius to achieve tight theoretical guarantees. Designing the uncertainty set for the model mismatch is typically straightforward, often relying on domain knowledge [39]. However, incorporating an additional penalty term to address data sparsity presents significant challenges. Balancing the conservativeness for less-visited state-action pairs estimation and the overall performance of learned policies requires careful consideration. In our work, we conduct a meticulous analysis to understand how the penalty radius impacts the performance of the learned policy. We then design penalty radii for three widely studied uncertainty set models: total variation, χ 2 divergence, and KL divergence. Our analysis provides insights into the performance of learned policies under these penalty radii, showcasing the versatility of our framework across both metricbased and non-metric-based models. Moreover, our designs enable us to derive tight theoretical guarantees. Specifically, we improve upon existing results for robust offline RL under the total variation model [5], match the state-of-the-art performance under the KL divergence model [36], and present the first result under the χ 2 divergence model.

2 Preliminaries

2.1 Markov Decision Process (MDP)

An MDP is specified by a tuple ( S , A , P , r, γ ) , where S and A are the state and action spaces, respectively, P = { P a s ∈ ∆ ( S ) , s ∈ S , a ∈ A } 1 is the transition kernel, r : S × A → [0 , 1] denotes the reward function, and γ ∈ [0 , 1) is the discount factor. Let S be the number of states and A be the number of actions. For any transition kernel P , P a s = ( p a s,s ′ ) s ′ ∈ S , where p a s,s ′ denotes the probability of transiting from state s to state s ′ after taking action a . The reward of the transition when taking action a at state s is denoted by r ( s, a ) .

For a stationary policy π that maps from state s ∈ S to a distribution over action a ∈ A , it specifies the probability of the agent taking actions at each state. The value function of a policy π is defined as

where E P denotes the expectation with respect to the distribution induced by the transition kernel P . Let ρ be the distribution of the initial state s , the value function under the distribution ρ is denoted by V π P ( ρ ) = E s ∼ ρ [ V π P ( s )] . Let d π P ( s ) and d π P ( s, a ) denote the state occupancy measure and state-action occupancy measure of policy π when the initial state s follows distribution ρ : d π P ( s ) = (1 -γ ) ∑ ∞ t =0 γ t P ( s t = s | s 0 ∼ ρ, π, P ) , d π P ( s, a ) = d π P ( s ) π ( a | s ) .

2.2 Distributionally Robust MDP

A robust MDP is defined as ( S , A , P , r, γ ) , where the transition kernel is not fixed but lies in some uncertainty set P . In this work we consider the ( s, a ) -rectangular uncertainty set:

where ⊗ s,a means the uncertainty sets for every state-action pair are independently defined, P a s is some nominal transition kernel for ( s, a ) -pair, D is some function that measures the difference between two distributions, e.g., total variation and KL divergence, and R is the radius of the uncertainty set. The robust MDP aims to find the policy that optimizes the worst-case performance among all possible transition kernels from P . Such a worst-case performance can be characterized by the robust value function V π P ( s ) :

which is shown to be the unique solution to the robust Bellman equation [13]:

with σ P a s ( V ) ≜ min q ∈ P a s q ⊤ V being the support function of a vector V on the uncertainty set P a s .

Similar to the standard MDP setting, V π P ( ρ ) can also be defined w.r.t. the initial state distribution ρ . The goal here is then to find the optimal robust policy arg max π V π P ( ρ ) .

3 Problem Formulation

In offline RL, the agent does not receive new samples by interacting with the environment. Instead, it is given a previously collected dataset D which consists of N tuples { ( s i , a i , s ′ i , r i ) : i = 1 , · · · , N } . The dataset is generated according to some distribution µ , i.e., ( s i , a i ) ∼ µ , and s ′ i ∼ P a i s i follows the nominal transition kernel P , and r i = r ( s i , a i ) is a deterministic reward function. As discussed, due to the potential model mismatch between the environment that generates the offline dataset and the one in which the learned policy is going to be deployed, we design an uncertainty set that

captures such a model mismatch as in (2), aiming to learn a policy that performs well under the true deployment environment through the robust RL framework:

where R represents the similarities of the two environments. D and R are generally designed by domain experts, so we do not focus on their design and consider them as pre-settled in this work.

A key challenge here is that the nominal transition kernel P 0 is unknown, but only an offline dataset generated from P 0 is given. Due to the limited exploration and distributional shift, the dataset D may not cover all possible states or transitions that the agent might encounter in the environment.

To guarantee that a provable efficient algorithm can be designed based on the dataset D , we adopt an assumption on the distributional mismatch between the dataset distribution and the occupancy measure induced by a comparator policy π ∗ . A comparator policy refers to a policy having satisfying worst-case performance and is set to be the optimal robust policy in many cases, but our approach can be applied to an arbitrary policy that may not be optimal.

Assumption 1. (Robust single-policy clipped concentrability [36]). The data distribution µ satisfies

In Assumption 1, we only require that the dataset covers the state-action pairs that are visited by the comparator policy, known as partial coverage. When there is no model mismatch, Assumption 1 reduces to the single-policy clipped concentrability assumption in [19] for non-robust offline RL.

We hence formulate the offline RL problem with model mismatch as the following concrete problem: Solve the robust RL problem (5) using only a fixed offline dataset D satisfying Assumption 1.

4 Framework Design and Main Results

To simultaneously address the two sources of uncertainty (1) uncertainty arising from the model mismatch between the data-collected environment and the deployment environment; and (2) uncertainty in the nominal (data-collection) model estimation attributed to an insufficient amount of data and limited coverage of the offline dataset, we develop a unified principle of pessimism, and theoretically characterize the finite sample complexity of the proposed algorithms. Our algorithms are easier and more efficient to implement than existing approaches, and yield improved or matching results.

Denote N ( s, a ) = ∑ N i =1 1 ( s i ,a i )=( s,a ) as the number of samples that transit from ( s, a ) in D , where 1 is the indicator function. The empirical transition kernel is then obtained as

and we also define the empirical reward function as

The MDP ˆ M = ( S , A , ˆ P , ˆ r ) with the empirical transition kernel ˆ P and empirical reward function ˆ r is referred to as the empirical MDP, representing our estimation of the nominal model from the dataset. A straightforward approach is to construct an uncertainty set ˆ P by replacing the nominal kernel P in (5) by ˆ P :

and solve the corresponding robust RL problem. Although it tackles the model mismatch uncertainty through the uncertainty set, it generally results in a sub-optimal performance due to the lack of consideration of the estimation error within the empirical model.

To tackle this estimation error, previous works introduce some additional structures other than the above DRO framework (9), including a penalty term in reward [36] or an additional uncertainty set and making it a bi-level DRO [5] (more discussions and comparisons can be found in Section 5). In this work, we construct a single, unified uncertainty set of environments, showing that such an estimation error can be incorporated into the distribution uncertainty set and developing a unified principle of pessimism to address both uncertainties. Specifically, for any s ∈ S and a ∈ A , we set

where κ a s is a function (will be specified later) inversely proportional to N ( s, a ) that measures the degree of confidence when estimating the empirical transition kernel; and R accounts for the model mismatch. Intuitively, for the less-observed state-action pairs, κ a s becomes larger and we are less confident and more pessimistic when estimating the transition kernel ˆ P a s , and vice versa.

We claim and show later that the additional uncertainty or pessimism by enlarging the uncertainty set effectively tackles the uncertainty from the limited dataset, and the whole uncertainty set results in a conservative estimation of the worst-case performance. We then optimize the worst-case performance under this uncertainty set ˜ P , which can be efficiently solved through the standard robust dynamic programming approach [28, 13]. Our algorithm is presented as Algorithm 1. For the convenience of analysis, we modify the vanilla robust value iteration algorithm [13] by setting the output policy to select actions that occur in the dataset when there is a tie. Such an output policy always exists (shown in Lemma 11 in the Appendix) and enables us to derive a tighter analysis of the sub-optimality gap. It is also worth noting that for all the uncertainty set models we consider, our Algorithm 1 can be efficiently applied with a polynomial computational complexity and a linear convergence rate [13].

Algorithm 1 Robust Value Iteration for Offline RL with Model Mismatch

Estimate the empirical reward ˆ r and empirical uncertainty set P according to (8) and (10)

In the following sections, we consider three widely used uncertainty set models: total variation, χ 2 divergence, and KL divergence models. We show that, for all three models, the uncertainty from the dataset can be incorporated into the uncertainty set as an additional term in the radius, and with a carefully designed uncertainty set, the optimal robust policy ˜ π obtained has a near-optimal performance under the target uncertainty set P , i.e., our approach effectively and efficiently solves the offline RL problems under model mismatch.

4.1 Total Variation (TV)

We first consider the case of TV defined uncertainty set 2 . Specifically, the uncertainty set is constructed using D ( q, p ) = 1 2 ∥ q -p ∥ 1 in eq. (10). Note that when the radius is greater than 2, the defined uncertainty set reduces to the whole probability simplex ∆ ( S ) . Our first attempt is to set κ a s large enough such that the true uncertainty set P falls into the constructed one ˜ P , such that the robust value function of ˜ P lower bounds the true robust value function V π P , as a conservative estimation. We note that as long as the true nominal transition kernel P and the empirical transition kernel ˆ P are close: ∥ P a s -ˆ P a s ∥ ≤ κ a s holds with high probability, the triangle inequality implies P ⊆ ˜ P with high probability. Therefore, the optimal robust policy ˜ π provides a performance guarantee for the original problem (5). Following this approach, we can show the following results.

Theorem 1. Consider TV defined uncertainty set. For each state-action pair ( s, a ) , set κ a s = √ S log SA δ 2 N ( s,a ) . Then with probability at least 1 -2 δ , the output policy ˜ π of algorithm 1 satisfies that

where ˜ O notation absorbs universal constants and log terms.

Remark 1. To achieve an ϵ sub-optimality gap, a dataset of size N = ˜ O ( S 2 C π ∗ (1 -γ ) -4 ϵ -2 ) is required. This result matches the previous one in [5].

The construction above is based on distribution, to ensure the resulting uncertainty set ˜ P is larger than the target one P . However, such a distribution-based construction can result in an overly large uncertainty set and an overly pessimistic policy, as observed in [47, 19]. To improve, one observation is that we can design a smaller uncertainty set such that the resulting robust value function V π ˜ P approximately lower bounds the true robust value function, without using the distribution-based framework. We hence further design a novel uncertainty set that turns out to be less conservative (the intuition of such a design will be discussed in the next section).

Theorem 2. For any ( s, a ) , let κ a s = log SA δ N ( s,a ) . For the output policy ˜ π of algorithm 1, with probability at least 1 -4 δ , it holds that

Here, µ min ≜ min s,a { µ ( s, a ) : µ ( s, a ) > 0 } denotes the smallest non-zero entry of the distribution µ that generates the dataset.

Remark 2. Combining the two results, we showed that our approach can obtain an ϵ -optimal robust policy when the dataset is of size ˜ O ( C π ∗ S (1 -γ ) 3 ϵ 2 min { S 1 -γ , 1 µ min } ) for the TV defined model. This result is better than the previous result in [5], illustrating our approach is more efficient. Moreover, our framework is much simpler than the one in [5] and can be effectively solved in a polynomial time (see detailed discussion in Section 5).

4.2 χ 2 Divergence

We then study the uncertainty models with the χ 2 divergence: D ( p, q ) = ∑ s q ( s ) ( 1 -p ( s ) q ( s ) ) 2 . Note that χ 2 divergence is not a metric, implying the failure of the triangle inequality and the distributionbased design as in the previous section. While it is possible to design κ such that D ( ˆ P a s , P a s ) ≤ κ a s [31], it does not imply P a s ⊂ ˜ P a s and no lower bound guarantee can be obtained.

To address this issue, we similarly adapt the value function-based construction, by taking a closer look at the error decomposition. The main idea of our distribution-based construction for the TV defined model is to construct an uncertainty set that is large enough to include the true transition kernel with high probability. Then the sub-optimality gap can be decomposed as

and ∆ 2 ≤ 0 from P a s ⊂ ˜ P a s , which however fails under non-metric models. On the other hand, if we further decompose ∆ 2 as

where ˆ P is the empirical uncertainty set (9). We note that ∆ 22 is the concentration error due to the limited dataset, which is independent of the term κ a s (ignoring the dependence of ˜ π on κ for discussion

connivance); ∆ 21 will be a negative term since ˆ P ⊂ ˜ P , thus we should set the term κ such that the negative bound on ∆ 21 cancels out with the concentration bound on ∆ 22 , leading to a non-zero yet tight overall bound on ∆ 2 . Instead of ensuring the uncertainty set inclusion, we directly ensure the bound on the value function difference ∆ 2 is small. Clearly, the clue function-based design can be applied to both non-metric and metric models, and is less conservative than the distribution-based for the metric models, since closeness in distribution is stronger than the closeness in value functions [47]. This observation results in our second design for the total variation model in Theorem 2, showing an improvement in sample complexity.

Based on this intuition, we present our radius design and results.

Theorem 3. Consider χ 2 divergence-defined uncertainty set. For any ( s, a ) , let κ a s = ˜ O (√ 1+ R N ( s,a ) ) When N ≥ ˜ O ( S C π ∗ µ 2 min ) , with probability at least 1 -4 δ , the output policy ˜ π of algorithm 1 satisfies

.

Remark 3. To achieve an ϵ -optimal robust policy under the χ 2 model, our approach requires the total number of samples

Our sample complexity has two parts: the ϵ -dependent part which dominates as the accuracy ϵ decreases, and a fixed amount of burn-in cost, whose existence is because we cannot expect to learn a near-optimal policy when the dataset is too limited. When the desired accuracy ϵ decreases, the first term dominates the overall complexity, resulting in the asymptotic result presented in the theorem.

Our approach and result stand for the first concrete study for offline robust RL with χ 2 divergencedefined uncertainty sets. It is also worth noting that our sample complexity asymptotically matches the sample complexity of the model-based robust RL with a generative model [38]. These observations hence demonstrate the optimality of our results and the effectiveness of our approach.

4.3 KL Divergence

In this section, we consider the KL divergence defined uncertainty set, i.e., D ( p, q ) = ∑ s p ( s ) log p ( s ) q ( s ) . Similarly, KL divergence is not a metric, we hence adapt our design discussed above for χ 2 models here. The following theorem presents our design of the uncertainty set and sample complexity results.

Theorem 4. Consider KL divergence defined uncertainty set. For any ( s, a ) , let κ a s = C 1 √ log 2(1+ R ) N 3 S (1 -γ ) δ N ( s,a ) ˆ P min , if N ≥ 8 log 1 δ µ min P min , then with probability at least 1 -4 δ , the output policy ˜ π of algorithm 1 satisfies

Here, P min and ˆ P min represents the minimal non-zero entry of the nominal transition kernel P and empirical nominal kernel ˆ P .

Remark 4. Similarly, for the KL divergence model, our approach requires a total number of ˜ O ( C π ∗ S (1 -γ ) 4 P min ϵ 2 + 1 P min µ min ) samples to find an ϵ -optimal policy. The sample complexity also contains two parts, an asymptotically dominated term and a fixed burn-in cost. Our result matches the one of [36] and is better than the one of [5].

To summarize, our unified framework can be adapted to different uncertainty models and solve offline robust RL problems, offering improved or matched sample complexity results and a more efficient

implementation. More discussion can be found in Section 5. We also provide some numerical experiments to verify the effectiveness and efficiency of our algorithm, which can be found in Appendix A. Under all three uncertainty set models, our algorithm enjoys a smaller or similar sample complexity compared to LCB approaches, and always outperform the non-robust dynamic programming approach.

5 Related Works

5.1 Comparison with prior art

In this section, we compare our works with the most related existing works [36, 5], where offline RL with model mismatch is studied. Compared to them, our methods enjoy three major advantages: (1). A more unified and straightforward framework of single pessimism principle; (2). Improved or matched sample complexity; And (3). Enhanced computational complexity.

Unified framework for double pessimism principles. In [36], the two principles of pessimism are separately employed, where a reward penalty term b is used to penalize less visited state-action pairs in addition to the uncertainty set that accounts for the model mismatch, and the update rule is

Our approach, on the other hand, enjoys a straightforward and simple formulation. Specifically, we incorporate the two principles of pessimism into a single uncertainty set, developing our unified principle. Moreover, it is noted that design of the penalty term b is complicated, whereas our addition term κ has a simple and clear form. It is worth noting that although the LCB approach and ours share a similar updating rule, there is a fundamental difference in the algorithm design motivation and analysis. In the LCB approach, the penalty term b is designed such that the resulting estimation V is less than the true robust value function V π P , to ensure the conservativeness of LCB approaches; Whereas our resulting estimation V π ˜ P is not necessarily less than V π P , since we directly tackle the distribution uncertainty but not through value function estimations. This hence requires novel technique innovations in our analysis.

In another closely related work [5], a double pessimism principle is adopted to address the two sources of uncertainty, under TV and KL divergence models. Specifically, a TV distance-based uncertainty set ˆ P is first constructed to tackle the model estimation uncertainty from the dataset; Then centered at each transition kernel ˆ P ∈ ˆ P , a second layer of uncertainty set Φ ( ˆ P ) to reflect the distributional robustness to model mismatch. They then take the optimal policy of

as the output policy. Although their approach indicates that the data estimation uncertainty can also be captured by a distributional uncertainty set, they still employ the two principles separately, although both in the form of DRO formulations. Instead of designing two uncertainty sets as in [5], we unify the two types of pessimism and use a single uncertainty set with a composed design of the radius to address both the model estimation uncertainty and the model mismatch.

Improved or Matched Sample Complexity. In terms of sample complexity, the comparison are included in Table 1. Compared to [36], our theoretical result matches theirs under the KL divergence model, and we moreover develop results for the other two uncertainty set models that are not considered therein. The numerical experiment results can be found in Appendix A, which further verify our discussion. Compared with [5], our results achieve better sample complexity in both KL and TV models. In the TV model, our sample complexity outperforms [5] in terms of dependence on S and (1 -γ ) ; For the KL model, our complexity is linearly dependent on S , while [5] has a quadratic dependence. Furthermore, as noted in [5], their result's exponential term can be replaced by utilizing both P min and µ min , while our (asymptotic) complexity result depends solely on P min .

Enhanced Computational Complexity. Our algorithms are also better in terms of computational complexity or practical implementations than both baselines.

The two-layer optimization problem in [5] is an extension of the model studied in [42] to the robust setting, both involving non-rectangular uncertainty sets that are NP-hard to solve [50]. This creates

uncertainty regarding the solvability of their models. Specifically, due to the unsolvability of the non-robust model in [42], an adversarial training-based algorithm is designed in [34] with only an experimental convergence guarantee, highlighting the implementation challenges of [5]. In contrast, our algorithm can be implemented with polynomial complexity. Specifically, the total computational complexity of our algorithm in the TV, CS and KL models are O ( S 2 A log S ) , O ( S 2 A log S ) , and ˜ O ( S 2 A ) .

Compared to [36], our algorithms also offer better computational complexity. Specifically, the penalty term in [36] requires a complicated computation involving a minimum operator, resulting in an additional max-min structure in their algorithm update. The comparison/max-min operator in the LCB algorithm is executed SA times per step, significantly increasing computational complexity. In contrast, our algorithms have a simple structure and do not require additional operators, making them more computationally efficient. We also use numerical experiment to further illustrate our enhanced computational efficiency, in Appendix A.

5.2 Other related works

We then discuss some of the other related works.

Offline RL without model mismatch. Offline RL focuses on learning an optimal policy from a pre-collected dataset, and the target deployment environment is identical to the one where the dataset is collected. Many previous works make the global coverage assumption, i.e., the behavior policy can cover all state-action pairs, e.g., [35, 7, 27, 58, 59, 14, 45, 20, 21, 64, 41, 9, 52, 18, 2, 10]. This assumption is too restrictive and is often violated in practice since it requires the history data to cover all the state-action pairs [12, 1, 11]. Recently, a relaxed partial coverage setting was proposed, which assumes that the density ratio between the occupancy measure induced by a single target policy and the behavior policy is finite for all state-action pairs, and the goal is to learn a policy that is no worse than the target policy. The partial coverage assumption only requires that the history data visit the state-action pairs that the target policy will visit. Under the partial coverage assumption, optimal policy can be learned for offline RL incorporated with the pessimism principle facing the uncertainty, e.g., [15, 42, 51, 53, 33, 62, 60, 37, 19, 63, 47]. However in our setting, we also consider the potential model mismatch between the two environments, which possibly due to e.g., non-stationarity, heterogeneity and sim-to-real gap, and formulate the problem as offline RL under model mismatch.

Robust RL. Robust RL [13, 28, 54] aims to tackle the model mismatch in RL, by optimizing the worst-case performance over the uncertainty set. Existing works mainly focus on the online setting [48, 49, 46, 4, 8, 25, 23] or with a generative model [57, 55, 29, 38]. Studies for robust RL with an offline dataset, besides the two mentioned above [36, 5], are developed in recent works including [24, 44, 65, 30, 56, 26, 57]. These works either focus on the linear MDPs, or adapt strong assumptions including global coverage or absorbing states. More importantly, all these works employ the two pessimism principles separately through the LCB penalty and DRO uncertainty set. Compared to them, we focus on general MDPs and develop our unified framework.

6 Conclusion and Discussions

In this paper, we investigated the offline RL problem under model mismatch under the most general partial coverage setting, where two sources of uncertainty are presented: inaccurate estimation of transition dynamics due to limited dataset coverage, and model mismatch between training and testing

environments. We developed a unified DRO-based framework containing a single uncertainty set with a composed radius of two parts to tackle the two sources of uncertainty discussed above. Our approach can be implemented in a much easier and more straightforward way than existing approaches and can be applied to both metric-based and non-metric-based uncertainty models. Specifically, we investigated three types of uncertainty sets defined by total variation, χ 2 divergence, and KL divergence. Our methodology can be easily extended to handle other uncertainty set models. Among them, we obtain near-optimal sample complexity results that improve or match the existing results under the total variation and KL divergence models and provide the first algorithm and finite sample complexity analysis for the uncertainty set defined by the χ 2 divergence.

Limitations. It is in our future interest to extend our unified framework to address large-scale problems. This includes robust MDPs with latent structures, such as linear MDPs, as opposed to previous work that uses the LCB + DRO framework, for example [24, 44], and more general MDPs with function approximation techniques.

References

A. Agarwal, S. Kakade, and L. F. Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory , pages 67-83. PMLR, 2020.

A. Antos, R. Munos, and C. Szepesvari. Fitted Q-iteration in continuous action-space mdps. In Neural Information Processing Systems , 2007.

M. A. Arcones. A bernstein-type inequality for u-statistics and u-processes. Statistics & probability letters , 22(3):239-247, 1995.

K. P. Badrinath and D. Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In Proc. International Conference on Machine Learning (ICML) , pages 511-520. PMLR, 2021.

J. Blanchet, M. Lu, T. Zhang, and H. Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659 , 2023.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540 , 2016.

J. Chen and N. Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

J. Dong, J. Li, B. Wang, and J. Zhang. Online policy optimization for robust mdp. arXiv preprint arXiv:2209.13841 , 2022.

Y. Duan, Z. Jia, and M. Wang. Minimax-optimal off-policy evaluation with linear function approximation. In Proc. International Conference on Machine Learning (ICML) , pages 27012709. PMLR, 2020.

A. M. Farahmand, R. Munos, and C. Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems , 2010.

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

C. Gulcehre, Z. Wang, A. Novikov, T. L. Paine, S. G. Colmenarejo, K. Zolna, R. Agarwal, J. Merel, D. Mankowitz, C. Paduraru, et al. RL unplugged: Benchmarks for offline reinforcement learning. arXiv preprint arXiv:2006.13888 , 2020.

G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257280, 2005.

N. Jiang. On value functions and the agent-environment boundary. arXiv preprint arXiv:1905.13341 , 2019.

Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096, 2021.

B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learning , pages 45-73. Springer, 2012.

S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

G. Li, L. Shi, Y. Chen, Y. Chi, and Y. Wei. Settling the sample complexity of model-based offline reinforcement learning. in preparation , 2022.

P. Liao, Z. Qi, and S. Murphy. Batch policy learning in average reward Markov decision processes. arXiv preprint arXiv:2007.11771 , 2020.

B. Liu, Q. Cai, Z. Yang, and Z. Wang. Neural trust region/proximal policy optimization attains globally optimal policy. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Z. Liu, Q. Bai, J. Blanchet, P. Dong, W. Xu, Z. Zhou, and Z. Zhou. Distributionally robust Q-learning. In International Conference on Machine Learning , pages 13623-13643. PMLR, 2022.

Z. Liu and P. Xu. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. arXiv preprint arXiv:2402.15399 , 2024.

Z. Liu and P. Xu. Minimax optimal and computationally efficient algorithms for distributionally robust offline reinforcement learning. arXiv preprint arXiv:2403.09621 , 2024.

M. Lu, H. Zhong, T. Zhang, and J. Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm. arXiv preprint arXiv:2404.03578 , 2024.

X. Ma, Z. Liang, J. Blanchet, M. Liu, L. Xia, J. Zhang, Q. Zhao, and Z. Zhou. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2209.06620 , 2022.

R. Munos. Error bounds for approximate value iteration. In Proceedings of the National Conference on Artificial Intelligence , volume 20, page 1006. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.

A. Nilim and L. El Ghaoui. Robustness in Markov decision problems with uncertain transition matrices. In Proc. Advances in Neural Information Processing Systems (NIPS) , pages 839-846, 2004.

K. Panaganti and D. Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics , pages 9582-9602. PMLR, 2022.

K. Panaganti, Z. Xu, D. Kalathil, and M. Ghavamzadeh. Robust reinforcement learning using offline data. arXiv preprint arXiv:2208.05129 , 2022.

K. Panaganti, Z. Xu, D. Kalathil, and M. Ghavamzadeh. Bridging distributionally robust learning and offline rl: An approach to mitigate distribution shift and partial data coverage. arXiv preprint arXiv:2310.18434 , 2023.

Y. Pitcan. A note on concentration inequalities for u-statistics. arXiv preprint arXiv:1712.06160 , 2017.

P. Rashidinejad, B. Zhu, C. Ma, J. Jiao, and S. Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

M. Rigter, B. Lacerda, and N. Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581 , 2022.

B. Scherrer. Approximate policy iteration schemes: A comparison. In Proc. International Conference on Machine Learning (ICML) , pages 1314-1322, 2014.

L. Shi and Y. Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 , 2022.

L. Shi, G. Li, Y. Wei, Y. Chen, and Y. Chi. Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. International Conference on Machine Learning , pages 19967-20025, 2022.

L. Shi, G. Li, Y. Wei, Y. Chen, M. Geist, and Y. Chi. The curious price of distributional robustness in reinforcement learning with a generative model. arXiv preprint arXiv:2305.16589 , 2023.

T. Siddique, J. L. Hau, S. Atallah, and M. Petrik. Robust pest management using reinforcement learning. The Multi-disciplinary Conference on Reinforcement Learning and Decision Making , 2019.

R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

M. Uehara, J. Huang, and N. Jiang. Minimax weight and Q-function learning for off-policy evaluation. In Proc. International Conference on Machine Learning (ICML) , pages 9659-9668. PMLR, 2020.

M. Uehara and W. Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 , 2021.

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

H. Wang, L. Shi, and Y. Chi. Sample complexity of offline distributionally robust linear markov decision processes. arXiv preprint arXiv:2403.12946 , 2024.

L. Wang, Q. Cai, Z. Yang, and Z. Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations , 2019.

Q. Wang, C. P. Ho, and M. Petrik. Policy gradient in robust mdps with global convergence guarantee, 2023.

Y. Wang, J. Xiong, and S. Zou. Achieving the minimax optimal sample complexity of offline reinforcement learning: A dro-based approach, 2023.

Y. Wang and S. Zou. Online robust reinforcement learning with model uncertainty. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 7193-7206, 2021.

Y. Wang and S. Zou. Policy gradient method for robust reinforcement learning. In Proc. International Conference on Machine Learning (ICML) , volume 162, pages 23484-23526. PMLR, 2022.

W. Wiesemann, D. Kuhn, and B. Rustem. Robust Markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. Bellman-consistent pessimism for offline reinforcement learning. arXiv preprint arXiv:2106.06926 , 2021.

T. Xie and N. Jiang. Batch value-function approximation with only realizability. arXiv preprint arXiv:2008.04990 , 2020.

T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

H. Xu and S. Mannor. Distributionally robust Markov decision processes. In Proc. Advances in Neural Information Processing Systems (NIPS) , pages 2505-2513, 2010.

Z. Xu, K. Panaganti, and D. Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 9728-9754. PMLR, 2023.

R. Yang, C. Bai, X. Ma, Z. Wang, C. Zhang, and L. Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems , 35:23851-23866, 2022.

W. Yang, L. Zhang, and Z. Zhang. Towards theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. arXiv preprint arXiv:2105.03863 , 2021.

M. Yin, Y. Bai, and Y.-X. Wang. Near-optimal offline reinforcement learning via double variance reduction. arXiv preprint arXiv:2102.01748 , 2021.

M. Yin and Y.-X. Wang. Optimal uniform ope and model-based offline reinforcement learning in time-homogeneous, reward-free and task-agnostic settings. arXiv preprint arXiv:2105.06029 , 2021.

M. Yin and Y.-X. Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34, 2021.

C. Yu, J. Liu, S. Nemati, and G. Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021.

A. Zanette, M. J. Wainwright, and E. Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34, 2021.

W. Zhan, B. Huang, A. Huang, N. Jiang, and J. D. Lee. Offline reinforcement learning with realizability and single-policy concentrability. arXiv preprint arXiv:2202.04634 , 2022.

J. Zhang, A. Koppel, A. S. Bedi, C. Szepesvari, and M. Wang. Variational policy gradient method for reinforcement learning with general utilities. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 4572-4583, 2020.

Z. Zhou, Q. Bai, Z. Zhou, L. Qiu, J. Blanchet, and P. Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In Proc. International Conference on Artifical Intelligence and Statistics (AISTATS) , pages 3331-3339. PMLR, 2021.

A Experiments

In this section, we provide simulation results to demonstrate the performance of our algorithm. We consider two problems: the Frozen-Lake problem [6] and the gambler problem [40, 65, 36].

For the Frozen-Lake problem, an agent aims to cross a 4 × 4 frozen lake from Start to Goal without falling into any Holes. The reward is set to be 1 when the agent reaches Goal and 0 otherwise. Due to the slippery nature of the frozen lake, the agent may not always move along the intended direction.

We formulate the gambler problem as an infinite-horizon MDP. A gambler engages in a betting game based on a sequence of coin flips. The gambler wins the stake when the coin lands on heads and loses it when the coin lands on tails. The probability of heads for the coin flip is p = 0 . 6 . The game begins with an initial balance and ends when the gambler's balance either reaches 20 or 0 . The reward is set to be 1 when the state reaches 20 and 0 otherwise.

A.1 Comparison under the total variation uncertainty set model

We first evaluate our algorithm under the total variation uncertainty set. We adapt the construction in [36] to design an DRVI-LCB approach for the TV defined uncertainty set as a baseline, and implement the two variants of our algorithm: distribution-based design and value function-based design. The robustness level R is set to be 0 . 1 . We generate the dataset according to µ ( s, a ) = 1 a = π ∗ ( s ) 2 + 1 a = η 2 where η is a random action, and π ∗ ( s ) denotes the optimal robust policy. Clearly the dataset satisfies the partial coverage assumption 1.

We run the algorithms independently for 10 times and plot the mean of the sub-optimality gap and mean plus and minus the standard deviations of the 10 runs as the envelop. It can be seen from Fig.1 and Fig. 2 that the value function-based construction has a smaller sub-optimality gap and converges faster than the distribution-based construction and the DRVI-LCB, which demonstrates that our algorithm is less conservative and more effective.

Value function based

0.8-

Distribution based

DRVI-LCB

0.6

de6Aewdo-qns

0.4

0.2

F0°0

0

200

400

600

800

1000

1200

1400

Size of dataset

Value function based

1.2

Distribution based

DRVI-LCB

1.0

Sub-optimality gap

0.8

0.6

0.4

0.2

0

200

400

600

800

1000

1200

1400

Size of dataset

A.2 Comparison under the χ 2 divergence uncertainty set model

We then compare our algorithm with the DRVI-LCB under the χ 2 divergence model. Similarly, we adapt their penalty term design for the uncertainty set, and run the experiment under the two environments. We similarly generate dataset following µ constructed above and run the algorithm for 10 times. The robustness level R is set to be 0 . 1 . It can be seen from Fig.3 and Fig.4 that the non-robust DP converges much slower to the optimal policy, whereas our approach has a similar convergence rate to the DRVI-LCB to the optimal policy, which validates our theoretical results.

Robust DP

Non-robust DP

1.0

DRVI-LCB

Sub-optimality gap

0.8

0.6

0.4

0.2

0.0

0

200

400

600

800

1000

1200

1400

Size of dataset

3.0

Robust DP

Non-robust DP

2.5

DRVI-LCB

Sub-optimality gap

2.0

1.5

1.0

0.5

0.0

0.5

0

100

200

300

400

500

600

700

Size of dataset

A.3 Comparison under the KL divergence uncertainty set model

We then compare the performance of our algorithm and the one in [36] under the KL divergence model. We similarly generate dataset following µ constructed above and run the algorithm for 10 times. The robustness level R is set to be 0 . 1 . It can be seen from Fig.5 and Fig.6 that the non-robust DP converges much slower to the optimal policy, whereas our approach has a similar convergence rate to the DRVI-LCB to the optimal policy, which validates our theoretical results.

0.8

Robust DP

Non-robust DP

DRVI-LCB

0.6

de6 Klewdo-qns

0.4

0.2

F0°0

0

200

400

600

800

1000

1200

1400

Size of dataset

3.0

Robust DP

Non-robust DP

2.5

DRVI-LCB

2.0

1.5

1.0

0.5

0.0

0.5

0

100

200

300

400

500

600

700

Size of dataset

We further run the three algorithms on the two uncertainty sets for the gambler problem. The robustness level R is set to be 0 . 2 . From Fig. 4 and Fig. 6, it can be seen that the non-robust DP converges much slower, whereas our approach solves the problem efficiently. Compared to LCB approaches, our method enjoys a similar performance and convergence rate, further demonstrating the effectiveness and efficiency of our approach.

A.4 Execution time

To illustrate our computational efficiency, we implemented the LCB algorithm from [36] and our DRO algorithm under the KL model, monitoring the execution time of both methods while learning the same policy from the same dataset. We plotted the execution time for each dataset versus the size of the dataset in three environments: Gambler's game, Frozen Lake, and N-chain. As shown in Appendix A.4, our algorithm consistently requires less execution time across all three environments, demonstrating lower computational complexity.

Gambler Game:KL-Divergence

N-Chain: KL-Divergence

DRO

45

DRO

18

DRO

12

LCB

40

LCB

LCB

16

35

Execution time (-s)

10

Execution time (-s)

14

30

Execution time (

8

25

12

20

10

6

15

8

10

6

0

100

200

300

400

500

600

700

0

100

200

300

400

500

600

700

800

0

100

200

OOE

400

500

600

700

800

Dataset size

Dataset size

Dataset size

B Proofs of Section 4.1

Lemma 1. For each state-action pair ( s, a ) , set

Then with probability at least 1 -δ , P a s ⊆ ˜ P a s .

We then present the sub-optimality gap in the following theorem.

Proof. For the ( s, a ) pairs with N ( s, a ) = 0 , the statement is trivial due to the fact that ˜ P a s = ∆ ( S ) . It is hence sufficient to consider the pairs with N ( s, a ) > 0 .

By directly applying the Hoeffding's inequality [22], we have that for each pair ( s, a, x ) ,

It hence can be further shown that

Proof. First note that and hence

simultaneously for any ( s, a ) -pair with probability at least 1 -δ .

Now consider any q ∈ P a s , it holds that ∥ q -P a s ∥ ≤ R . Hence

which implies that P a s ⊆ ˜ P a s for any ( s, a ) with probability at least 1 -δ .

Theorem 5. For the output policy ˜ π of algorithm 1, with probability at least 1 -2 δ , it holds that

Due to the fact that P a s ⊆ ˜ P a s with probability 1 -δ , it holds that

It can be further bounded as

where ( a ) is from ˜ π = arg max π V π ˜ P , and the last inequality is from the fact P ˜ V ∈ P ∗ s and hence σ ∗ s ( V π ∗ P ) ≤ ( P ˜ V ) ∗ s V π ∗ P .

Applying the inequality above recursively implies

Note that

where the last inequality is from

where ( a ) is from the dual form of σ P ( V ) and ( b ) is from the fact that | max x F ( x ) -max x G ( x ) | ≤ max x | F ( x ) -G ( x ) | .

From the definition of κ a s = √ S log SA δ 2 N ( s,a ) , it holds that κ ∗ s = √ S log SA δ 2 N ( s,π ∗ ( s )) ≤ √ 8 S log 4 SA δ log SA δ 2 Nµ ( s,π ∗ ( s )) and hence

To further bound (28), we first note that

which is from the dual form of σ P ( V ) , i.e.,

and [ V ] λ ( s ) = min { V ( s ) , λ } . We then involve (20), and it holds that

Plugging this inequality to (27) and from the definition of C π ∗ , we have that

which completes the proof.

Theorem 6. For the output policy ˜ π of algorithm 1, with probability at least 1 -4 δ , it holds that

Proof. Note that Lemma 8 of [36] states that with probability 1 -δ , for any ( s, a ) pair,

We hence conduct our proof under the occurrence of this event. Note that for any s ,

From Lemma 2 and Lemma 3, the above bound can be bounded as

which completes the proof.

Lemma 2. (Bound on Term A ) With probability at least 1 -2 δ , it holds that

Proof. To bound term ( A ) , note that

where ( a ) is from ˜ π = arg max π V π ˜ P , and the last inequality is from the fact P ˜ V ∈ P ∗ s and hence σ ∗ s ( V π ∗ P ) ≤ ( P ˜ V ) ∗ s V π ∗ P .

Applying (40) recursively implies

Note that

where the last inequality is from

where ( a ) is from the dual form of σ P ( V ) and ( b ) is from the fact that | max x F ( x ) -max x G ( x ) | ≤ max x | F ( x ) -G ( x ) | .

Since N ( s, a ) ≥ Nµ ( s,a ) 8 log 4 SA δ , from Assumption 1, we have that

From the definition of κ a s = 1 N ( s,a ) , it holds that κ ∗ s = 1 N ( s,π ∗ ( s )) ≤ 8 log 4 SA δ Nµ ( s,π ∗ ( s )) ≤ 8 C π ∗ log 4 SA δ N ( S + 1 d ∗ P ˜ V ( s,π ∗ ( s )) ) and hence

To further bound (42), we first note that

which is from the dual form of σ P ( V ) , i.e.,

and [ V ] λ ( s ) = min { V ( s ) , λ } . We then utilize Lemma 15 and (44), and it holds that

We then bound the two terms as follows.

Bound on Term A 1

We first claim the following inequality:

To prove (49), we note that

and hence for any s ∈ S ,

which proves (49).

Now with (49), we first note that

where the last inequality is due to the fact ∥ V ˜ π ˜ P + γ P ∗ ˜ V V ˜ π ˜ P ∥ ≤ 2 1 -γ and (49). We first have that

where ( a ) is from (52), ( b ) is due to γ < 1 , ( c ) is from the definition of visitation distribution.

Note that by Cauchy's inequality, ∑ s d ∗ P ˜ V ( s ) √ Var ( P ˜ V ) ∗ s ( V ˜ π ˜ P ) ≤ √∑ s d ∗ P ˜ V ( s ) Var ( P ˜ V ) ∗ s ( V ˜ π ˜ P ) , hence

where the first inequality is from (44) and the last inequality follows similarly as (42). Hence, Term A 1 can be bounded as

Bound on Term A 2

From Lemma 10, it is straightforward to see that for any transition kernel q a s ∈ P a s ,

Hence

Thus Term A 2 can be bounded as

Combine the bounds we obtained for terms A 1 and A 2 in (55) and (58), we have that

which further implies that

Then, combine (41), (42) and the inequality above, we have that

which completes the proof.

Lemma 3. (Bound on Term B ) If N ≥ 196 c 1 log 4 SAN δ log 4 SA δ (1 -γ ) 2 Nµ min , then with probability at least 1 -2 δ , it holds that

Proof. From the robust Bellman equation, it holds that

where ( P V ) ˜ π ( s ) s = arg min q ∈ P ˜ π ( s ) s qV ˜ π P , and ( P ˜ V ) ˜ π ( s ) s = arg min q ∈ P ˜ π ( s ) s qV ˜ π ˜ P , and the last inequality is from ( P V ) ˜ π ( s ) s ∈ P π ∗ ( s ) s and ( P ˜ V ) ˜ π ( s ) s V ˜ π ˜ P = σ P ˜ π ( s ) s ( V ˜ π ˜ P ) ≤ ( P V ) ˜ π ( s ) s V ˜ π ˜ P . On the other hand,

where ( a ) is from the fact that P ˜ V ∈ P and hence σ ˜ π s ( V ˜ π P ) ≤ ( P ˜ V ) ˜ π ( s ) s V ˜ π P .

Hence we have that

We then bound terms ( I ) and ( II ) . Note that the only difference between the two terms are the visitation distributions, i.e, d ˜ π P V and d ˜ π P ˜ V .

Bound on Term ( I )

We first note that

hence it is sufficient to bound B 1 , B 2 .

To bound B 1 , note that

where ( a ) is from the dual form of the support function [13] and the last inequality is because Span ( V ) ≤ 1 1 -γ for any 0 ≤ V ≤ 1 1 -γ .

We then bound B 2 . Similarly from the dual form, it holds that

We then apply Lemma 15, it holds that

with probability at least 1 -δ and K 1 = ( 2 + c 2 log ( 4 SAN 2 δ ) + √ 2 c 1 log ( 4 SAN 2 δ ) ) .

Note that for any s, a ,

Firstly, Lemma 7 of [38] and Lemma 10 imply that

To bound √ | Var ( P V ) a s ( V ˜ π ˜ P ) -Var ( P V ) a s ( V ˜ π P ) | , note that it holds that | Var ( P V ) a s ( V ˜ π ˜ P ) -Var ( P V ) a s ( V ˜ π P ) | ≤ ∥ V ˜ π ˜ P -V ˜ π P ∥ 2 , thus

And the last term can be bounded as

using (56), and hence

Hence by combining (72),(73) and (75) we have that

Thus plug the inequality above and the bound of B 1 in (63), we have that

Bound on Term II

Similarly to the analysis in bounding the Term (I), we have that

where B 1 can be bounded as

The bound on B 2 similarly follows as in Term ( I ) . Note that

with probability at least 1 -δ and K 1 = ( 2 + c 2 log ( 4 SAN 2 δ ) + √ 2 c 1 log ( 4 SAN 2 δ ) ) .

Note that for any s, a ,

Hence

Similarly, Lemma 13 of [38] and Lemma 10 imply that

And to bound √ Var P a s ( V ˜ π ˜ P ) -Var ( P ˜ V ) a s ( V ˜ π ˜ P ) , note that

from (56), and hence

Hence by combining (83) and (85) we have that

Thus plug the inequality above and the bound of B 1 , we have that

Then we combine the bounds of terms ( I ) and ( II ) together, and we have that

Due to the fact that N ≥ 196 c 1 log 4 SAN δ log 4 SA δ (1 -γ ) 2 Nµ min ,

and hence completes the proof.

C Proofs of Section 4.2

Theorem 7. With probability at least 1 -4 δ , the output policy ˜ π of algorithm 1 satisfies

Proof. Using the similar decomposition implies that

The proof is then completed by combing the following two lemmas.

T

Lemma 4. (Bound on Term A ) With probability at least 1 -4 δ , it holds that

Proof. To bound term ( A ) , note that

where ( a ) is from ˜ π = arg max π V π ˜ P , and the last inequality is from the fact P ˜ V ∈ P ∗ s and hence σ ∗ s ( V π ∗ P ) ≤ ( P ˜ V ) ∗ s V π ∗ P .

Applying (93) recursively implies

Note that

The first term in (95) can be bounded as follows. Recall the dual form of the support function w.r.t. the uncertainty set P = { q ∈ ∆ ( S ) : χ 2 ( q || p ) ≤ R } as follows:

where V α ( s ) = min { V ( s ) , α } .

Applying the dual form further implies that

where the last inequality directly follows from the Hoeffding's inequality [38] and 1 N -net technique used in Lemma 15.

As for the second term in (97), we utilize Lemma 7 and the technique of ϵ -net, which implies that

This hence bounds the first term in (95). For the second term in (95), similarly apply the dual form and we have that

where the last inequality is from the fact that Var ( V α ) ≤ 1 (1 -γ ) 2 for any V ≤ 1 1 -γ and any α ≤ 1 1 -γ . We now plug (98) and (99) to (95) and (94), and we have that

where the last inequality is from (44). This hence completes the proof.

Lemma 5. (Bound on Term B ) With probability at least 1 -4 δ , it holds that

Proof. Similarly to the proof of Lemma 4, note that

where ( ˜ P V ) ˜ π ( s ) s = arg min q ∈ ˜ P ˜ π ( s ) s qV ˜ π P . We then bound the term ˜ σ ˜ π ( s ) s ( V ˜ π P ) -σ ˜ π ( s ) s ( V ˜ π P ) .

From Lemma 6, it holds that which completes the proof.

Lemma 6. With probability at least 1 -δ , it holds that for any s ∈ S ,

Proof. We first consider a vector V that is independent from ˆ P .

where α ∗ ≜ arg max α { ˆ P ˜ π ( s ) s ( V α ) -√ ( R + κ ˜ π ( s ) s ) Var ˆ P ˜ π ( s ) s ( V α ) } , and the last inequality is from the fact that -max f ≤ -f ( x ) for any x .

We then construct an ϵ 1 -net over [0 , 1 1 -γ ] , such that there exists β with ∥ V α ∗ -V β ∥ ≤ ϵ 1 . Then we further have that

where we use the Bernstein inequality and the technique of ϵ 1 -net. To bound the last two terms, we note that

where we again use the ϵ 1 -net technique in ( a ) , and apply Lemma 7 in ( b ) . Combining (106) and (107) implies that

Clearly, if we set C = √ c 2 + RC 1 and κ ˜ π ( s ) s = C N ( s, ˜ π ( s )) +2 √ CR N ( s, ˜ π ( s )) = ˜ O (√ 1 N ( s, ˜ π ( s )) ) , (109) then ( √ c 2 N ( s, ˜ π ( s )) + √ RC 1 N ( s, ˜ π ( s )) + √ R -√ R + κ ˜ π ( s ) s ) ≤ 0 , and hence ˜ σ ˜ π ( s ) s ( V ) -σ ˜ π ( s ) s ( V ) ≤ c 1 (1 -γ ) N ( s, ˜ π ( s )) + ϵ 1 (1 -γ ) + √ ϵ 1 (1 -γ ) + C 2 √ R (1 -γ ) N ( s, ˜ π ( s )) ≤ ˜ O ( √ 1 + R (1 -γ ) N ( s, ˜ π ( s )) ) , (110)

by setting ϵ 1 ≤ ˜ O ( 1 N ) .

Now to show the claim for V ˜ π ˜ P , we construct an ϵ 2 -net over [0 , 1 1 -γ ] S . By applying the similar trick, we complete the proof.

Lemma 7. For any vector V ∈ [0 , 1 1 -γ ] that is independent with P , with probability at least 1 -δ , it holds that

Proof. This result can be derived from the Bernstein inequality for U -statistics [3, 32], by noting that the sample standard deviation √ n n -1 Var ˆ P a s ( V ) is an U -statistics and

D Proofs of Section 4.3

Theorem 8. If N ≥ 8 log 1 δ µ min , then there exists some universal constants C 1 , C 2 , such that with probability at least 1 -4 δ , it holds that

Proof. Note that

The proof completes by combining the bounds on the two terms, shown in the following two lemmas.

Lemma 8. There exists a constant C 2 , such that with probability at least 1 -δ , it holds that

Proof. From the definition,

hence

where q a s is the worst-case transition kernel of V ˜ π ˜ P in P a s , and ( a ) is from σ P π ∗ s ( V π ∗ P ) -σ P π ∗ s ( V ˜ π ˜ P ) = σ P π ∗ s ( V π ∗ P ) -( q π ∗ s )( V ˜ π ˜ P ) ≤ ( q π ∗ s )( V π ∗ P -V ˜ π ˜ P ) , and c ( s ) ≜ γσ P π ∗ s ( V ˜ π ˜ P ) -γσ ˜ P π ∗ s ( V ˜ π ˜ P ) .

Recursively applying (117) further implies that

We moreover rewrite c as

Moreover, we introduce two sets

For s ∈ S 1 , (216) of [36] implies that d π ∗ q ( s ) = 0 .

We then focus on s ∈ S 2 . It has been shown in [36] that µ ( s, π ∗ ( s )) > 0 and N ( s, π ∗ ( s )) ≥ N min { 1 S ,d π ∗ q ( s ) } 12 C π ∗ .

Thus Lemma 17 first implies that

To bound ∆ 1 , 2 , note that

where ( a ) is from the dual solution of KL-divergence, and ( b ) is due to the fact that the optimal solutions to both dual forms satisfy λ ∗ ≤ 1 (1 -γ )( R + κ π ∗ ( s ) s ) [65], and ( c ) is due to max f -max g ≤ max | f -g | .

Then we plug in the definition of κ π ∗ ( s ) s = C 1 √ log 2(1+ R ) N 3 S (1 -γ ) δ N ( s,π ∗ ( s )) min x ˆ P π ∗ s,x and using (202) of [36], we have that

Combine (122) and (124), we have that

Thus we have that

where ( a ) is from Cauchy inequality, and ( b ) is from (220) of [36], which hence completes the proof.

We then bound the term ∆ 2 .

Lemma 9.

Proof. First, note that

where q ˜ π s is the worst-case transition kernel of V ˜ π P in P ˜ π s , and b ( s ) ≜ γ ( σ ˜ P ˜ π s ( V ˜ π ˜ P ) -σ P ˜ π s ( V ˜ π ˜ P )) . Recursively applying (128) implies

We further introduce two sets as follows.

For s ∈ S 1 , ˜ P ˜ π s = ∆ ( S ) , hence

For s ∈ S 2 , we have that

Hence invoke Lemma 17, we have that for s ∈ S 2 , with probability at least 1 -δ ,

To further bound the RHS of (134), we first note that Lemma 8 of [36] states that if Nµ ( s, a ) ≥ 8 log 1 δ , then with probability 1 -δ , for any ( s, a ) pair,

This moreover implies that with probability 1 -δ , for s ∈ S 2 ,

On the other hand, (202) of [36] states that with probability at least 1 -δ ,

Hence by plugging (136) and (137) in (134) we have that

On the other hand, similarly to (123), it holds that

where the last inequality is from the fact that max F -max G ≥ F ( x ) -G ( x ) , ∀ x . Combining with(134) further implies that

by combining (138) and (139). Thus

Hence combine with (129) and (132), we further have that

E Auxiliary Lemmas

Lemma 10. It holds that

Proof. Note that when N > 8 log 4 SA δ (1 -R ) µ min and the fact that N ( s, a ) ≥ 8 log 4 SA δ Nµ min , it holds that

Denote s ∗ = arg min s V ˜ π ˜ P ( s ) . Then it holds that

Wedenote the optimal action ˜ π ( s ) by a in the following proof. Note that there exists a vector q a s ∈ R S , such that ˆ P a s ≥ q a s ≥ 0 , and ∑ s ′ q a s ( s ′ ) = 1 -R -κ a s . This is doable because ∑ s ′ ˆ P a s ( s ′ ) = 1 and R + κ a s ≤ 1 . Hence it implies that the transition kernel q a s + ( R + κ a s ) 1 s ∗ ∈ ˜ P a s , since ∥ q a s +( R + κ a s ) 1 s ∗ -ˆ P a s ∥ ≤ ∥ q a s -ˆ P a s ∥ +( R + κ a s ) ≤ 2( R + κ a s ) .

Hence

where the last equation is from ∑ s ′ q a s ( s ′ ) = 1 -R -κ a s and ∥ V ˜ π ˜ P ∥ = max s V ˜ π ˜ P = V ˜ π ˜ P ( s ∗ ) . Plug this inequality in (145), and it holds that

Thus

which implies that

Lemma 11. Recall the set S 0 ≜ { s ∈ S : N ( s ) = 0 }

(1). For any policy π and s ∈ S 0 , V π ˜ P ( s ) = 0

. Then ;

(2). There exists a deterministic robust optimal policy ˜ π , such that for any s / ∈ S 0 , N ( s, ˜ π ( s )) > 0 .

Proof. Proof of (1).

For any s ∈ S 0 , it holds that N ( s, a ) = 0 for any a ∈ A . Hence ˆ r ( s, a ) = 0 and ˜ P a s = ∆ ( S ) . Then for any policy π and a ∈ A , it holds that

which implies V π ˜ P ( s ) = 0 together with the fact that V π ˜ P ≥ 0 .

Proof of (2).

We prove Claim (2) by contradiction. Assume that for any optimal policy ˜ π , there exists s / ∈ S 0 such that N ( s, ˜ π ( s )) = 0 . We then consider a fixed pair ( ˜ π, s ).

N ( s, ˜ π ( s )) = 0 further implies ˆ r ( s, ˜ π ( s )) = 0 , ˜ P ˜ π ( s ) s = ∆ ( S ) , and

where the last inequality is from ˜ P ˜ π ( s ) s = ∆ ( S ) , ˆ r ( s, ˜ π ( s )) = 0 , and σ ˜ P ˜ π ( s ) s ( V ˜ π ˜ P ) ≤ 1 s V ˜ π ˜ P = V ˜ π ˜ P ( s ) . This further implies that V ˜ π ˜ P ( s ) = 0 because V ˜ π ˜ P ≥ 0 .

̸

which is contradict to V ˜ π ˜ P ( s ) = max a Q ˜ π ˜ P ( s, a ) = Q ˜ π ˜ P ( s, ˜ π ( s )) .

̸

(II). If r ( s, b ) = 0 , Lemma 12 then implies the modified policy f s b (˜ π ) is also optimal, and satisfies N ( x, f s b (˜ π )( x )) = N ( x, ˜ π ( x )) for any x = s , and N ( s, f s b (˜ π )( s )) > 0 .

Then consider the modified policy f s b (˜ π ) .

̸

If there still exists s ′ / ∈ S 0 such that N ( s ′ , f s b (˜ π )( s ′ )) = 0 , then similarly, there exists another action b ′ = f s b (˜ π )( s ′ ) such that N ( s ′ , b ′ ) > 0 . Then whether r ( s ′ , b ′ ) > 0 , which falls into Case (I) and leads to a contradiction, or applying Lemma 12 again implies another optimal policy f s ′ b ′ ( f s b (˜ π )) , such that N ( s, f s ′ b ′ ( f s b (˜ π ))( x )) = N ( s, f s b (˜ π )( x )) > 0 for x / ∈ { s, s ′ } , N ( s, f s ′ b ′ ( f s b (˜ π ))( s )) = N ( s, f s b (˜ π )( s )) > 0 and N ( s ′ , f s ′ b ′ ( f s b (˜ π ))( s ′ )) > 0 .

Repeating this procedure recursively further implies there exists an optimal policy π , such that N ( s, π ( s )) > 0 for any s / ∈ S 0 , which is a contraction to our assumption.

Therefore it completes the proof.

Lemma 12. For a robust optimal policy ˜ π , if there exists a state s / ∈ S 0 and an action b such that N ( s, ˜ π ( s )) = 0 , r ( s, b ) = 0 and N ( s, b ) > 0 , define a modified policy f s b (˜ π ) as

̸

Then the modified policy f s b (˜ π ) is also optimal, and satisfies N ( s, f s b (˜ π )( s )) > 0 , N ( x, f s b (˜ π )( x )) = N ( x, ˜ π ( x )) , ∀ x = s .

̸

Proof. Recall that ˜ P ˜ π ( s ) s = ∆ ( S ) and ˜ P b s ⊆ ∆ ( S ) , we have that

On the other hand, since s / ∈ S 0 , there exists another action b = ˜ π ( s ) such that N ( s, b ) > 0 , and hence ˆ r ( s, b ) = r ( s, b ) . We consider the following two cases.

(I). If r ( s, b ) > 0 , then

Thus

where ˜ P b s is a modified uncertainty set defined as

̸

Now we have that

which further implies V f s b (˜ π ) ˜ P b s ( s ) = 0 . Note that in eq. (152), we have shown V ˜ π ˜ P ( s ) = 0 , hence V f s b (˜ π ) ˜ P b s ( s ) = V ˜ π ˜ P ( s ) = 0 .

̸

Now consider the two robust Bellman operator T s b V ( x ) = ∑ a f s b (˜ π )( a | x )(ˆ r ( x, a ) + γσ ( ˜ P b s ) a x ( V )) and T V ( x ) = ˆ r ( x, ˜ π ( x )) + γσ ˜ P ˜ π ( x ) x ( V ) . It is known that V f s b (˜ π ) ˜ P b s is the unique fixed point of the robust Bellman operator T s b and V ˜ π ˜ P is the unique fixed point of T . When x = s ,

̸

where ( a ) is from f s b (˜ π )( x ) = ˜ π ( x ) when x = s , ( b ) is from ( ˜ P b s ) ˜ π ( x ) x = ˜ P ˜ π ( x ) x . And for s , it holds that

where ( a ) is from ( ˜ P b s ) b s = ∆ ( S ) and ˆ r ( s, b ) = r ( s, b ) = 0 = ˆ r ( s, ˜ π ( s )) , and ( b ) follows from the fact ˜ P ˜ π ( s ) s = ∆ ( S ) .

eq. (160) and eq. (161) further imply that V ˜ π ˜ P is also a fixed point of T s b . Hence it must be identical to V f s b (˜ π ) ˜ P b s , i.e., V f s b (˜ π ) ˜ P b s = V ˜ π ˜ P .

Combine with eq. (156), we have

̸

which implies that f s b (˜ π ) is also optimal with N ( s, f s b (˜ π )( s )) = N ( s, b ) > 0 . And since f s b (˜ π )( x ) = ˜ π ( x ) for x = s , then N ( x, f s b (˜ π )( x )) = N ( x, ˜ π ( x )) . This thus completes the proof.

Lemma 13 (Lemma 4, [19]) . For any δ , with probability 1 -δ , max { 12 N ( s, a ) , 8 log NS δ } ≥ Nµ ( s, a ) , ∀ s, a .

Lemma 14 (Lemma 9, [19]) . For any ( s, a ) pair with N ( s, a ) > 0 , if V is an vector independent of ˆ P a s obeying ∥ V ∥ ≤ 1 1 -γ , then with probability at least 1 -δ ,

Lemma 15. For the total variation uncertain set and for any fixed s, a and δ , with probability at least 1 -δ , it holds that

and and

For the state s , note that

Proof. Step 1: Construct an auxiliary robust MDP. For the state s ∈ S and any constant u ∈ [0 , 1] , we construct the following MDP M s,u = ( S , A , r s,u , P s,u ) with transition kernel P s,u and reward r s,u :

̸

̸

Centered at M s,u , we further construct the following robust MDP ˜ M s,u = ( S , A , ˜ P s,u , r s,u ) :

̸

The robust Bellman operator associated with this robust MDP is hence

and we denote the robust value function w.r.t. it by V s,u .

Step 2: Prove V ˜ π ˜ P is a robust value function w.r.t. ˜ M s,u ∗ for some u ∗ ∈ [0 , 1] . We claim that if we set u ∗ = (1 -γ ) V ˜ π ˜ P ( s ) ∈ [0 , 1] , then the robust value function w.r.t. ˜ M s,u ∗ is equal to V ˜ π ˜ P .

To prove the claim, recall that V ˜ π ˜ P is the unique fixed point of the optimal robust Bellman operator

(173)

We further have that

̸

which is due to the construction of ˜ M s,u ∗ ; And for states x = s , we have that

which is from the construction of ˜ M s,u ∗ and the fact that V ˜ π ˜ P is the fixed point of ˜ T .

These two equations hence imply that V ˜ π ˜ P is a fixed point of the robust Bellman operator T s,u ∗ , which further proves the claim.

Step 3: Decouple the dependence of V ˜ π ˜ P on ˆ P by constructing an 1 N -net.

Define U = { i N : i = 0 , 1 , ..., N } . Clearly, U is a 1 N -net [43] of the interval [0 , 1] , i.e., for any u ∈ [0 , 1] , there exists u i ∈ U , such that | u -u i | ≤ 1 N .

Clearly, each u ∈ U is a constant independent with ˆ P a s , hence applying Lemma 14 implies that for any u ∈ U ,

Take the union bound over any ( s, a, u ) ∈ S × A × U , and we have the following inequality holds with probability at least 1 -δ :

Step 4: Approximate | P a s V ˜ π ˜ P -ˆ P a s V ˜ π ˜ P | using the 1 N -net.

As we showed above, there exists u ∈ U such that | u -u ∗ | ≤ 1 N . Moreover, note that

where the last inequality is from σ ( ˜ P s,u ) a x ( V s,u ∗ ) = σ ( ˜ P s,u ∗ ) a s ( V s,u ∗ ) for any x, a . Hence we have that

where the last inequality is from

Thus (182) can be further bounded as

We note that this inequality exactly matches (127) in [38], hence the following part directly follows, which is omitted here.

Lemma 16. (Lemma 14 of [36]) For any ( s, a ) satisfying N ( s, a ) > 0 , and any vector V ∈ R | S | independent of ˆ P a s obeying ∥ V ∥ ≤ 1 1 -γ , with probability at least 1 -δ , it holds that

Lemma 17. For the KL-divergence uncertainty set, and for any ( s, a ) satisfying N ( s, a ) > 0 , with probability at least 1 -δ , it holds that

Proof. Step 1. We first construct a robust MDP with state-absorbing empirical nominal transition kernels. More specifically, for each state s and a constant u ≥ 0 , define the following transition kernels for any a ∈ A , x ∈ S :

̸

Moreover, define the modified reward function:

̸

We then define the following uncertainty set ( P s,u ) a x for any a ∈ A , x ∈ S as follows:

̸

The auxiliary robust MDP is then defined as M s,u = ( P s,u = ⊗ s,a ( P s,u ) a s , r s,u ) .

Step 2. We next show that if we set u ∗ = (1 -γ ) V ˜ π ˜ P ( s ) , then the optimal robust value function of M s,u ∗ is identical to V ˜ π ˜ P .

̸

For any state x = s and action b , it can be verified from the definitions that ( P s,u ) b x = ˜ P b x , and hence

For state s , we have that

Thus we verified that V ˜ π ˜ P is the fixed point of the robust Bellman operator of M s,u , and hence identical to the optimal robust value function of M s,u .

Step 3. Define the set U ≜ { i N : 1 ≤ i ≤ N } . The set U is then a 1 N -net of the interval [0 , 1] . Since it is clear that u ∗ ≤ 1 , there exists u 0 ∈ U , such that | u 0 -u ∗ | ≤ 1 N .

On the other hand, for any u ∈ U , since u ≤ 1 , the optimal robust value function ∥ V s,u ∥ ≤ 1 1 -γ ; Moreover, from the construction, the uncertainty set P s,u is independent of ˆ P a s , hence invoking Lemma 16 implies

with probability at least 1 -δ .

Step 4. We further show the following claim:

To show (196), for s , we have that

where ( a ) is from | max f -max g | ≤ max | f -g | , and ( b ) is from the fact that ( P s,u ) b s = { 1 s } for any u and b .

̸

And for state x = s , we have that

̸

where ( a ) is from ( P s,u ) b x = ˜ P b x and r s,u 0 ( x, b ) = r s,u ∗ ( x, b ) for any b ∈ A and x = s .

Hence together we proved (196), which is identical to (240) of [36]. The remaining proof hence follows exactly the same as [36].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the claims are directly from the main theorems of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations and future works in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs of all the theorems are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Information that are needed to reproduce the main experimental results of the paper is disclosed in Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code will be released if the paper is accepted. The experiments are straightforward and easy to reproduce.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We clearly specify all the details for the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot a error bar/envelop to indicate the statistical significance. For 10 independence experiments, we plot the mean and the standard deviation of them as the error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: No computing resources are required.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reciewed the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no potential societal impact that needs to be specified.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm that tackles the offline RL task (for tabular state and action space) under two key challenges (i) the mismatch between the environment dynamics used to generate the dataset and the environment used to run the learned policy (for evaluation), and (ii) the bias in the data-collection procedure where for certain (s, a) pairs there are only a few samples in the dataset. In contrast to prior approaches that tackle the two problems separately, this paper proposes a unified solution to both problems under the distributionally robust MDP framework. \n\nIntuitively, distributionally robust MDP aims to learn a policy/value function with the best possible expected return given the worst possible environment dynamics in a set of distributions. Specifically, the set contains all distributions that are close (i.e., smaller than R) under some metrics (this paper considered total variation, chi-squared divergence, and KL divergence). This paper also takes into account the scarcity of data for certain (s, a) pairs by changing the fixed bound R to an adaptive bound R + K_{s}^{a}, where K_{s}^{a} depends on the visit count of (s, a) in the offline dataset. \n\nWith this new formulation, the authors derive high-probability lower bounds of the optimality of the learned value functions, under three divergences: total variation, chi-squared divergence, and KL divergence). The bounds appear to be tighter than that in prior work.\n\nSTRENGTHS:\nThis paper proposes a theoretically useful offline RL algorithm that can handle model misspecification errors and data scarcity in a unified perspective. Specifically, it turns the task into a canonical algorithm for distributionally robust MDP. This can significantly simplify the analysis and inspire future work to find better connections between the two challenges in offline RL.\n\nThe paper contains a thorough introduction and comparison with prior work, which makes it much easier to see the contributions of this paper.\n\nWEAKNESSES:\nOne major weakness of the proposed algorithm is its practicality. The proposed algorithm relies on tabular state and action representations and involves solving robust Bellman equations which could be prohibitively hard in practice. Also, to compute certain statistics, we need to record the state-action visitation count, which is impractical in practice.\n\nIt would be nice if the author can demonstrate potential ways to apply the algorithm in practice. For example, can the algorithm be used together with function approximations?\n\nQUESTIONS:\nDoes the proposed method achieve better empirical performance on tabular environments compared to baselines?\n\nAre there potential ways to make the algorithm more practical?\n\nLIMITATIONS:\nThe authors discussed certain limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The authors propose a unified principle of pessimism using distributionally robust Markov decision processes (MDPs) to handle both data sparsity and model mismatch. They construct a robust MDP with a single uncertainty set and demonstrate that the optimal robust policy achieves a near-optimal sub-optimality gap under the target environment across three uncertainty models: total variation, χ2 divergence, and KL divergence. The proposed approach improves or matches state-of-the-art performance under these models and provides the first result for the χ2 divergence model.

STRENGTHS:
- The paper is well-structured and clearly written, making it easy to follow the main arguments and methodologies.
- The paper provides detailed theoretical analysis and guarantees, showing that the proposed method achieves near-optimal performance.
- The proposed framework is versatile as well as easier to implement.

WEAKNESSES:
- While the theoretical guarantees are strong, the empirical validation could be more comprehensive. The experiments are conducted on relatively simple tasks (Frozen-Lake and Gambler problems). It would be beneficial to see results on more complex and diverse benchmarks.
- Computational Cost: The paper claims that the proposed method has a lower computational cost compared to existing methods, but this claim is not empirically validated. A comparison of computational costs (e.g., runtime) with other methods would strengthen this claim.

QUESTIONS:
To summarize the main points/questions raised in the weaknesses section:

- Could the authors provide empirical results on more complex tasks to validate the effectiveness of their method?
- Could the authors compare the computational costs of their method with other existing methods?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline reinforcement learning (RL) under model mismatch. The authors propose a unified distributionally robust optimization (DRO) framework that effectively tackles both the uncertainty from limited dataset coverage and the model mismatch between the training and deployment environments.\n\nSTRENGTHS:\nA Unified DRO Formulation: The authors provide unified results for robust Markov Decision Processes (MDPs) with several types of robust sets.\n\nThe authors have carefully designed the uncertainty set radius, incorporating both model mismatch and data sparsity factors. The tight theoretical guarantees provided for three widely studied uncertainty set models (total variation, χ2 divergence, and Kullback-Leibler divergence) demonstrate the robustness and versatility of their approach.\n\nThe authors present the first finite-sample analysis for the χ2 divergence uncertainty set in the offline reinforcement learning (RL) setting with model mismatch. This contribution advances our understanding of this particular model and its applications in robust RL.\n\nThe paper is well-structured, with a clear problem formulation, comprehensive algorithmic description, and rigorous theoretical analysis.\n\nWEAKNESSES:\nIt would be beneficial if you could elaborate on the advantages of your work compared to [35] and [5], especially regarding sample complexity under KL and total variation settings. A more detailed comparison would help readers fully appreciate the advancements your method brings to the field.\n \nGiven that [5] presents another unified framework for offline distributionally robust RL, it would be interesting to know if your framework can be extended to the χ2 case. If this extension is not possible, highlighting this limitation could actually strengthen your paper by emphasizing the unique contributions of your approach.\n\nAddressing the computational efficiency of your proposed algorithm, particularly in comparison to [35], would be valuable. If your method is computationally efficient, including experimental results similar to those in [35] could provide empirical support for your theoretical findings.\n\nQUESTIONS:\nna\n\nLIMITATIONS:\nThe authors have discussed the limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}