Title: Percentile Criterion Optimization in Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Bayesian Uncertainty

Robustness
Abstract: In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the percentile criterion. The percentile criterion is optimized by constructing an uncertainty set that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing these sets itself is challenging. Existing works use Bayesian credible regions as uncertainty sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any uncertainty sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller uncertainty sets and learns less-conservative robust policies.

Percentile Criterion Optimization in Offline Reinforcement Learning

Abstract

In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the percentile criterion . The percentile criterion is approximately solved by constructing an ambiguity set that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses Bayesian credible regions as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Valueat-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.

1 Introduction

Batch Reinforcement Learning (Batch RL) [26] is popularly used for solving sequential decisionmaking problems using limited data. These algorithms are crucial in high-stakes domains where exploration is either infeasible or expensive, and policies must be learned from limited data. In model-based Batch RL algorithms, transition probabilities are learned from the data as well. Due to insufficient data, these transition probabilities are often imprecise. Errors in transition probabilities can accumulate, resulting in low-performing policies that fail when deployed.

To account for the uncertainty in transition probabilities, prior work uses Bayesian models [10, 13, 27, 40, 45, 47] to model uncertainty and optimize the policy to maximize the returns corresponding to the worst α -percentile transition probability model. These policies guarantee that the true expected returns will be at least as large as the optimal returns with high confidence. This technique is commonly referred to as the percentile-criterion optimization. Unfortunately, the percentile criterion is NP-hard to optimize. Thus, current work uses Robust Markov Decision Processes (RMDPs) to optimize a lower bound on the percentile criterion. An RMDP takes as input an ambiguity set (uncertainty set) that contains the true transition probability model with high confidence and finds a policy that maximizes the returns of the worst model in the ambiguity set.

Since the percentile criterion is non-convex, constructing ambiguity sets itself is a challenging problem. Existing work uses Bayesian credible regions (BCR) [40] as ambiguity sets. However, these ambiguity sets are often unnecessarily large [15, 40] and result in learning conservative robust policies.

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Some recent works approximate ambiguity sets using various heuristics [3, 40], but we show that they remain too conservative. Thus, the question of the optimal ambiguity set, i.e., ambiguity sets that result in optimizing the tightest possible lower bound on the percentile criterion and less-conservative policies remains unanswered.

Our Contributions In this paper, we answer two important questions: a) Are Bayesian credible regions the optimal ambiguity sets for optimizing the percentile criterion? b) Can we obtain a less conservative solution to the percentile criterion than RMDPs with BCR ambiguity sets while retaining its percentile guarantees? Our theoretical findings show that Bayesian credible regions can grow significantly with the number of states and therefore, tend to be unnecessarily large, resulting in highly conservative policies. As our main contribution, we provide a dynamic programming framework (Section 3), which we name the VaR framework, for optimizing a lower bound on the percentile criterion without explicitly constructing ambiguity sets. Specifically, we propose a new robust Bellman operator, the Value at Risk ( VaR ) Bellman operator, for optimizing the percentile criterion. We show that it is a valid contraction mapping that optimizes a tighter lower bound on the percentile criterion, compared to RMDPs with BCR ambiguity sets (Section 3). We theoretically analyze and bound the performance loss of our framework (Section 3.1). We provide a Generalized VaR Value iteration algorithm and analyze its error bounds. We also show that there exist directions in which the Bayesian credible regions can grow unnecessarily large with the number of states in the MDP and possibly result in a conservative solution. On the other hand, the ambiguity sets implicitly optimized by the VaR Bellman operator tend to be smaller, i.e., they have a smaller asymptotic radius and are independent of the number of states (Section 4). Finally, we empirically demonstrate the efficacy of our framework in three domains (Section 5).

1.1 Related Work

Several work[13, 34, 40] propose different methods for solving the percentile criterion, as well as other robust measures for handling uncertainty in the transition probabilities estimates. Russel and Petrik [40] and Behzadian et al. [3] propose various heuristics for minimizing the size of the ambiguity sets constructed for the percentile-criterion. Russel and Petrik [40] propose a method that interleaves robust value iteration with ambiguity set size optimization. Behzadian et al. [3] propose an iterative algorithm that optimizes the weights of ℓ 1 and ℓ ∞ ambiguity sets while optimizing the robust policy. However, these methods still construct Bayesian credible sets which can be unnecessarily large and result in conservative policies, as we show in Section 5.

Other works consider partial correlations between uncertain transition probabilities to mitigate the conservativeness of learned policies [4, 15, 19, 29, 30]. These approaches mitigate the conservativeness of S- and SA-rectangular ambiguity sets by capturing correlations between the uncertainty and by limiting the number of times the uncertain parameters deviate from the mean parameters. Despite these heuristics, most of these works [2, 20, 40, 53] either rely on weak statistical concentration bounds to construct frequentist ambiguity sets, or use Bayesian credible regions as ambiguity sets. These sets still tend to be unnecessarily large [15, 40], resulting in conservative policies.

Finally, a large number of works [12, 16, 36, 44, 49, 50, 52] have proposed RL algorithms that use measures like Conditional Value at Risk, Entropic risk measure amongst other risk measures. However, we note that these works use risk measures to obtain robustness guarantees against aleatoric uncertainty (system uncertainty) and not epistemic uncertainty (model uncertainty), which is the focus of our work. Since these works optimize a completely different objective, we do not compare our framework against theirs. Robust RL work [2, 14, 20, 28, 32, 55] proposes other robust measures for handling uncertainty in transition probabilities; however, these approaches do not provide probabilistic guarantees on the expected returns, and compute overly conservative policies.

2 Preliminaries

In the standard reinforcement learning setting, a sequential decision task is modeled as a Markov Decision Process (MDP) [37, 48]. An MDP is a tuple ⟨S , A , P, R, p 0 , γ ⟩ that consists of (a) a set of states S = { 1 , 2 , . . . , S } , (b) a set of actions A = { 1 , 2 , . . . , A } , (c) a deterministic reward function R : S × A × S → R , (d) a transition probability function P : S × A → ∆ S , (e) an initial state distribution p 0 ∈ ∆ S , where ∆ S represents the S -dimensional probability simplex, and (f) a

discount factor γ ∈ [0 , 1] . We use p s,a to denote the vector of transition probabilities P ( s, a, · ) corresponding to the state s and action a . Likewise, we use r s,a to denote the vector of rewards R ( s, a, · ) corresponding to state s and action a . A Markovian policy π : S → ∆ A maps each state s to a distribution over actions A . In a general RL setting, the goal is to compute a policy π that maximizes the expected discounted return ρ ( π, P ) over an infinite horizon,

The value of a policy π at any state s is the discounted sum of rewards received by an RL agent, if it starts from state s , i.e., v π ( s ) = E [ ∑ ∞ t =0 γ t r ( s t , a t , s t +1 ) | s 0 = s, a t ∼ π, s t +1 ∼ P ( s t , a t , · )] .

We assume a batch reinforcement learning setting [26] where the reward function is known, but the true transition probabilities P ∗ are unknown. Following prior work on robust Bayesian RL [8, 13, 40, 54], we use parametric Bayesian models to represent uncertainty over the true transition probabilities P ∗ . We will use ˜ P = { ˜ p s,a } s ∈S ,a ∈A to denote the random transition probabilities. We assume we have a batch of data D = { s i , a i , s ′ i } N i =1 , which in conjunction with some prior distribution defines the posterior distribution of transition probabilities ˜ P .

The Fisher information measures the amount of information about the unknown true parameters θ ∗ ∈ R d carried by an observable random variable ˜ X ∈ R m . If f ( x ; θ ∗ ) is the probability density of ˜ X conditioned on θ ∗ , then the Fisher information is given by I ( θ ∗ )= E [ ( ∇ log f ( ˜ X ; θ ∗ ))( ∇ log f ( ˜ X ; θ ∗ )) T ] .

Then, in the Bayesian setting, the Bernstein von Mises theorem [51] states that under mild regularity conditions the posterior distribution of the parameters ˜ θ converges in the limit to the distribution of the MLEof θ ∗ which is asymptotically Gaussian, in particular, lim N →∞ √ N ( ˜ θ -θ ∗ ) ⇝ N ( 0 , I ( θ ∗ ) -1 ) where ⇝ indicates convergence in distribution. In many cases, the above holds even though the conditions of the Bernstein-von Mises theorem are not met [17, 18]. Of particular interest in this setting is the Dirichlet distribution where the asymptotic MLE is a multivariate Gaussian under certain conditions on the prior distribution [18], although in this case, it is degenerate (i.e., the covariance matrix is not full-rank).

For asymptotic results, we assume that the data D is sampled such that each state s and action a is observed infinitely many times as N → ∞ . Furthermore, we assume henceforth that the prior is asymptotically negligible, and the MLE is asymptotically Gaussian with covariance matrix I ( P ∗ ) -1 / N . Therefore, the posterior distribution of ˜ P is also asymptotically Gaussian with covariance matrix I ( P ∗ ) -1 / N . Under these conditions, we will estimate the Fisher information corresponding to θ ∗ = P ∗ using the asymptotic covariance of the posterior distribution of ˜ P .

To avoid unnecessary computational technicalities, we will assume that ˜ P is a discrete random variable taking on values ˜ P ( ω ) , ω ∈ Ω for some Ω = { 1 , . . . , M } with a distribution f . That is, the random variable ˜ P represents a discrete approximation of the true, possibly continuous posterior, as is common in methods like Sample Average Approximation (SAA) [43].

Percentile Criterion The α -percentile criterion is popularly used to derive robust policies under model uncertainty [13]. It aims to compute a policy π that maximizes the returns corresponding to the worst α -percentile model:

The value y lower-bounds the true expected discounted returns with confidence 1 -α where α ∈ (0 , 1 / 2 ) . Optimizing the percentile criterion is equivalent to optimizing the Value at Risk ( VaR α ) of expected discounted returns when there exists uncertainty in transition probabilities ˜ P and the expected returns function ρ is lower-semicontinuous. The optimization in (1) is equivalent to

where VaR α of a bounded random variable ˜ X with a CDF function F : R → [0 , 1] is defined as [38]

A lower value of α in (1) indicates a higher confidence in the returns achieved in expectation. For example, VaR 0 . 05 [ ρ ( π, ˜ P )] = x indicates that the true returns will be at least equal to the robust returns x for 95 % of the transition probability models. When clear from context, we use VaR to denote the Value at Risk at confidence level α . Unfortunately, the optimization problem in (1) is NP-hard to optimize and is usually approximately solved using Robust MDPs.

Robust MDPs Robust MDPs (RMDPs) generalize MDPs to account for uncertainty, or ambiguity, in the transition probabilities. An ambiguity set for an RMDP is constructed such that it contains the true model with high confidence. The optimal policy of a Robust MDP π ∗ maximizes the returns of the worst model in the ambiguity set: π ∗ = arg max π ∈ Π min P ∈P ρ ( π, P ) . General RMDPs are NP-hard to solve [53], but they are tractable for broad classes of ambiguity sets. The simplest such type is the SA-rectangular ambiguity set [33, 53], defined as

for a given P s,a ⊆ ∆ S , s ∈ S , a ∈ A . SA-rectangular ambiguity sets [3, 40] assume that the transition probabilities corresponding to each state-action pair are independent. Similarly to MDPs, the optimal robust value function v ∗ ∈ R S for an SA-rectangular RMDP is the unique fixed point of the robust Bellman optimality operator T : R S → R S defined as ( T v ) s = max a ∈A min p s,a ∈P s,a p T s,a ( r s,a + γ · v ) .

To optimize the percentile criterion, an SA-rectangular ambiguity set P is constructed such that it contains the true model with high probability, and thus, the following equation holds.

Although RMDPs have been used to solve the percentile criterion [3], the quality of the robust policies it computes depends mainly on the size of the ambiguity sets. The larger the ambiguity sets, the more conservative the robust policy [30]. SA-rectangular ambiguity sets are most commonly studied; thus we focus our attention on SA-rectangular Robust MDPs. We investigate whether Bayesian credible regions are optimal ambiguity sets for optimizing the percentile criterion. We refer to SA-rectangular RMDPs and SA-rectangular ambiguity sets as Robust MDPs and ambiguity sets respectively.

Our work focuses on Bayesian (rather than frequentist) ambiguity sets. Bayesian ambiguity sets are usually constructed from Bayesian credible regions (BCR) [3, 40]. Given a state s and an action a , let ψ s,a represent the size of the BCR ambiguity sets; P BCR α s,a and ¯ p s,a represent the mean transition probabilities. The set P BCR α s,a is constructed as

where q ∈ { 1 , ∞} represents the norm of the weighted ball in (4) and b ∈ R S + is a weight vector. Here, b is jointly optimized with ψ ∈ R to minimize the span of the ambiguity sets such that the true model is contained in the ambiguity set with high confidence, i.e., Pr(˜ p s,a ∈ P s,a ( b , ψ, q )) ≥ 1 -α . We refer to BCR ambiguity sets with non-uniform weights as weighted BCR ambiguity sets. We refer to the Robust Bellman optimality operator with BCR ambiguity sets ( T BCR α ) as the BCR α Bellman optimality operator, and to RMDPs with BCR ambiguity sets as BCR RMDPs. For any δ ∈ (0 , 1 / 2 ) , setting the confidence level α in T BCR α to δ / SA for all state-action pairs yields 1 -δ confidence on the returns of the optimal robust policy [3]. However, we show that even span-optimized BCR RMDPs can be sub-optimal for optimizing the percentile criterion.

We use the shorthand w s,a for any s ∈ S , a ∈ A to denote the vector of values associated with value v ∈ R S and the one-step return from state s and action a , i.e., w s,a = r s,a + γ v . We use ¯ p s,a ∈ R s and Σ s,a ∈ R S × S for any s ∈ S , a ∈ A to represent the empirical mean and covariance of transition probabilities ˜ p s,a estimated from D . We use tilde to indicate that it is a random variable. We use ϕ ( · ) and Φ( · ) to represent the probability distribution function (PDF) and cumulative distribution function (CDF) respectively of the normal distribution with mean 0 and variance 1 . The Z -Minkowski norm ∥ x ∥ Z for a vector x given some positive-definite matrix Z is defined as ∥ x ∥ Z = √ x T Z -1 x .

We illustrate the conservativeness of BCR α ambiguity sets with the following example.

Example 2.1 . Consider an MDP with four states { s 0 , s 1 , s 2 , s 3 } and a single action { a 0 } . The state s 0 is the initial state and the states s 1 , s 2 , s 3 are terminal states with zero rewards. For the

sake of simplicity, we assume that it is only possible to transition to state s 1 , s 2 and s 3 from state s 0 . The transition probability of p s 0 ,a 0 is uncertain and distributed as a Dirichlet distribution ˜ p s 0 ,a 0 ∼ Dir (10 , 10 , 1) with mean [0 . 48 , 0 . 48 , 0 . 04] . The rewards for transitions from state s 0 are given by r s 0 ,a 0 = [0 . 25 , 0 . 25 , -1] .

We wish to optimize the percentile criterion with confidence level δ = 0 . 2 . Following the sampling procedure proposed by Russel and Petrik [40] to construct a uniformly weighted BCR α ambiguity set for ˜ p s 0 ,a with 100 posterior samples, yields an ambiguity set P BCR α s 0 ,a 0 = { p ∈ ∆ S ∣ ∣ ∥ p -˜ p s 0 ,a 0 ∥ 1 ≤ 0 . 277 } . In this case, the reward estimate against the worst model in the ambiguity set p = [0 . 50 , 0 . 32 , 0 . 18] is ρ BCR α = 0 . 025 . Since we have a single non-terminating state in the MDP, the percentile returns are given by ρ VaR α = VaR 0 . 2 [ ˜ p T s 0 ,a 0 r s 0 ,a 0 ] . Computing ρ VaR α for Dirichlet distribution Dir (10 , 10 , 1) , we get ρ VaR α = 0 . 17 > ρ BCR α . Thus, this example shows that BCR ambiguity sets can be unnecessarily large, and thereby result in conservative policies.

3 VaR Framework

We introduce the VaR Bellman optimality operator T VaR α for approximately solving the percentile criterion. We show that T VaR α is a valid Bellman operator: it is a contraction mapping and lower bounds the percentile criterion. For any value function v ∈ R S , state s ∈ S and action a ∈ A , we define the VaR Bellman optimality operator T VaR α as

For each state s , T VaR α maximizes the value corresponding to the worst α -percentile model. In contrast to the BCR α Bellman optimality operator T BCR α , computing T VaR α does not require constructing ambiguity sets from confidence regions; it can simply be estimated from samples of the model posterior distribution, as we later show.

Proposition 3.1 (Validity) . The following properties of T VaR α hold for all value functions u , v ∈ R S .

The operator T VaR α is contraction mapping on R S : ∥T VaR α u -T VaR α v ∥ ∞ ≤ γ ∥ u -v ∥ ∞ .

The operator T VaR α is monotone: u ⪰ v ⇒T VaR α u ⪰ T VaR α v .

The equality T VaR α ˆ v = ˆ v has a unique solution.

Proposition 3.1 formally proves that T VaR α is a valid Bellman operator, i.e., it is a contraction mapping, monotone, and has a unique fixed point ˆ v = ( T VaR α ˆ v ) . Here on, we will refer to the policy ( ˆ π ) corresponding to the fixed point value ˆ v as the VaR α policy.

0.0

0.1

0.2

0.3

0.4

0.5

confidence level 1-

0

20

40

60

80

Ratio of asymptotic radii

S=100

S=200

S=300

S=400

S=500

(a) Asymptotic Radii

BCR

VaR

(b) Dirichlet(5,5,5)

BCR

VaR

(c) Dirichlet(30,30,30)

S2

S2

S3

S1

S3

We now show that the VaR Bellman optimality operator T VaR α optimizes a lower bound on the percentile criterion. Given a policy π , a state s ∈ S , and transition probabilities P , let

represent the Bellman evaluation operator for transition probabilities P . Furthermore, let

represent the VaR Bellman evaluation operator for random transition probabilities ˜ P . We use ˆ v π to denote the fixed point of T π VaR α . Furthermore, we use ˜ v π to represent the random fixed point of T π ˜ P , which is computed using a random realization of the transition probabilities ˜ P sampled from the posterior distribution conditioned on observed transitions D .

Proposition 3.2 (Lower Bound Percentile Criterion) . For any δ ∈ (0 , 1 / 2 ) , if we set the confidence level α in the operator T π VaR α to δ / S , then for every policy π ∈ Π : Pr ˜ P [ˆ v π ⪯ ˜ v π |D ] ≥ 1 -δ .

Proposition 3.2 shows that for any policy π and state s , the VaR α value at state s , ˆ v π ( s ) lower bounds the true value ˜ v π ( s ) with high confidence. Comparing Proposition 3.2 with the definition of the percentile-criterion in (1), we see that the percentile-criterion requires confidence guarantees only on the returns computed from the initial states, whereas the equation in Proposition 3.2 provides confidence guarantees on the value of every state. Therefore, for any policy π , the value p T 0 ˆ v π is a lower bound on the percentile-criterion objective VaR δ [ ρ ( π, ˜ P )] . Since T VaR α finds a policy π that maximizes the value p T 0 ˆ v π , it follows [37] that T VaR α optimizes a lower bound on the percentile criterion in (1).

Proposition 3.3. Suppose that ˜ p s,a for any state s and action a , is a multivariate subGaussian with mean ¯ p s,a and covariance factor Σ s,a , i.e., E [ exp ( λ ( ˜ p s,a -¯ p s,a ) T w ) ] ≤ exp ( λ 2 w T Σ s,a w / 2 ) , ∀ λ ∈ R , ∀ w ∈ R S . Then, for any state s ∈ S , T VaR α satisfies

As a special case, when ˜ p s,a is normally distributed ˜ p s,a ∼ N ( ¯ p s,a , Σ s,a ) , then T VaR α for any state s ∈ S can be expressed as

Proposition 3.3 shows that by assuming that the transition probabilities are sub-Gaussian, we can easily compute a lower bound of the VaR Bellman update ( T VaR α v ) for a given value function using only the mean and the covariance matrix of ˜ P . In the special case where ˜ P is normally distributed, we can compute the VaR Bellman optimality operator T VaR α ( v ) exactly.

3.1 Performance Guarantees

We now derive finite-sample and asymptotic bounds on the loss of the VaR framework.

Theorem 3.4 (Performance) . Let ˆ v be the fixed point of the VaR Bellman optimality operator T VaR α and π ∗ be the optimal policy in (1) . Let ρ ∗ = VaR δ [ ρ ( π ∗ , ˜ P )] denote the optimal percentile returns and ˆ ρ = p T 0 ˆ v denote the lower bound on the percentile returns computed using the Bellman operator T VaR α with α = δ / S . Then for each δ ∈ (0 , 1 / 2 ) :

Theorem 3.4 bounds the finite sample performance loss of the VaR framework. The loss varies proportionally to the maximum difference between the δ / S and 1 -( 1 -δ / S ) percentile of the one-step Bellman update for the optimal robust value function ˆ v . Furthermore, the VaR framework performs better when the uncertainty in the transition probabilities is small.

Theorem 3.5 (Asymptotic Performance) . Suppose that the normality assumptions on the posterior distribution ˜ P in Section 2 are satisfied. For any δ ∈ (0 , 1 / 2 ) , set α = δ / S in T VaR α . For any state s and action a , let I ( p ∗ s,a ) be the Fisher information matrix corresponding to the true transition probabilities p ∗ s,a . Furthermore, let σ 2 max = max s ∈S ,a ∈A ˆ w T s,a I ( p ∗ s,a ) -1 ˆ w s,a be the maximum over state-action pairs of the asymptotic variance of the returns estimate ˜ p T s,a ˆ w s,a . Then the asymptotic performance of the VaR framework ˆ ρ w.r.t. the optimal percentile returns ρ ∗ satisfies

Theorem 3.5 shows that the asymptotic loss in performance of the VaR framework convergence to 0, i.e., lim N →∞ ( ρ ∗ -ˆ ρ ) = 0 .

3.2 Dynamic Programming Algorithm

We provide a detailed description of the VaR value iteration algorithm (Algorithm 3.1) below. We also bound the number of samples required to estimate a single VaR Bellman update ( T π VaR α v ) s for any given policy π and state s with high confidence 1 -ζ .

Algorithm 3.1: Generalized VaR Value Iteration Algorithm

In each iteration of Algorithm 3.1, we compute the one-step VaR Bellman update T VaR α ( v ) using the current value function v . When ˜ P is not normally distributed, we use the Quick Select algorithm [23] to efficiently compute the empirical estimate of the α -percentile of 1-step returns for any state s , action a , and value function v , i.e., ̂ VaR α [ ˜ p T s,a ( r s,a + γ v )] . On the other hand, when ˜ P is normally distributed, we compute the VaR Bellman optimality update T VaR α ( v ) using the empirical estimate of mean ( ¯ p s,a ) s ∈S ,a ∈A and covariance ( Σ s,a ) s ∈S ,a ∈A of the transition probabilities derived from the data D (Proposition 3.3). We repeat these steps until convergence.

Proposition 3.6 (Empirical Error Bound) . For any state s , action a and value function v , let ̂ VaR α [ ˜ p T s,a w s,a ] represent the empirical estimate of α -percentile of returns VaR α [ ˜ p T s,a w s,a ] and Φ f represent the cumulative density function (CDF) of the random estimate of returns ˜ p T s,a w s,a . Suppose that Φ f is differentiable at the point VaR α [ ˜ p T s,a w s,a ] and let η = Φ ′ f ( VaR α [ ˜ p T s,a w s,a ] ) represents the density of estimate of returns at point VaR α [ ˜ p T s,a w s,a ] . Let M ∗ be the number of posterior samples required to obtain empirical error ε ∈ R , with confidence 1 -ζ , where 0 < ζ < 1 , i.e., Pr [ | ̂ VaR α [ ˜ p T s,a w s,a ] -VaR α [ ˜ p T s,a w s,a ] | > ε ] ≤ ζ . Then, lim ε → 0 M ∗ ε 2 = ln( 2 / ζ ) / 2 η 2 .

We now show that Algorithm 3.1 produces a policy π k and value function u k that approximates the optimal value function ˆ v .

Proposition 3.7 (Value Iteration Error) . Define the empirical VaR Bellman optimality operator T ̂ VaR α for any value v ∈ R S and state s ∈ S as ( T ̂ VaR α v ) s = max a ∈A ̂ VaR α [ ˜ p T s,a w s,a ] . Let

ˆ v ∈ R S and ˆ u ∈ R S represent the fixed points of T VaR α and T ̂ VaR α respectively. Suppose that Algorithm 3.1 returns policy π k and value function u k and ∥ ˆ u -u 0 ∥ ∞ ≤ r max / 1 -γ . Then,

Furthermore, the gap between the empirical and true value function is bounded by

Proposition 3.8 (Time Complexity) . Let r max = max s,s ′ ∈S ,a ∈A | R ( s, a, s ′ ) | . Then, Algorithm 3.1 terminates in k = ⌈ log 1 /γ ( rmax ε (1 -γ ) ) ⌉ iterations with time complexity O ( S 2 AM log 1 /γ ( rmax ε (1 -γ ) ) ) .

Furthermore, for any failure probability ζ ∈ (0 , 1) , suppose that the CDF ( Φ f ) of the random estimate of 1-step returns ˜ p T s,a ˆ w s,a is differentiable at the point VaR α [ ˜ p T s,a ˆ w s,a ] , and set η = Φ ′ (VaR α [ ˜ p T s,a ˆ w s,a ]) and M = O ( log ( S /ζ ) η 2 ε 2 (1 -γ ) 2 ) . Then with probability at least 1 -ζ , it holds that ∥ ˆ v -u k ∥ ∞ ≤ O ( ε ) , and Algorithm 3.1 runs in O ( S 2 A log 1 /γ ( rmax ε (1 -γ ) ) log ( S ζ ) η 2 ε 2 (1 -γ ) 2 ) time.

4 Comparison with Bayesian Credible Regions

We are now ready to answer the question: Are Bayesian credible regions the optimal ambiguity sets for optimizing the percentile criterion? For this, we compare the VaR framework with BCR Robust MDPs. First, we derive the robust form of the VaR framework and show that in contrast to the BCR Bellman operator, the VaR Bellman optimality operator implicitly constructs value function dependent ambiguity sets, and thus, these sets tend to be smaller (Proposition 4.1). Then, we compare the asymptotic radii of the BCR ambiguity sets and the VaR ambiguity sets implicitly constructed by T VaR α . For any given confidence level α , the radius of the VaR ambiguity sets are asymptotically smaller than that of BCR ambiguity sets (Theorem 4.3). Precisely, the ratio of the radii of VaR ambiguity sets to BCR ambiguity sets is at least √ χ 2 S -1 , 1 -α / Φ -1 (1 -α ) , where χ 2 S -1 , 1 -α is the CDF inverse of 1 -α percentile of Chi-squared distribution with degree of freedom S -1 and Φ -1 (1 -α ) is the 1 -α percentile of N (0 , 1) . This implies that there exist directions in which the BCR ambiguity sets are at least Ω( √ S ) larger than VaR ambiguity sets. Thus, we prove that VaR framework is better suited for optimizing the percentile criterion than RMDPs with BCR ambiguity sets.

For any value function v , define the VaR ambiguity set P VaR , v as

Proposition 4.1 (Equivalence) . The VaR Bellman optimality operator T VaR α can be expressed as

Furthermore, the optimal VaR policy ˆ π ∈ Π D solves max π ∈ Π D min P ∈P VaR , ˆ v ρ ( π, P ) , where ˆ v ∈ R S is the fixed point of the VaR Bellman operator T VaR α , i.e., ˆ v = ( T VaR α ˆ v ) .

Proposition 4.1 shows that the VaR Bellman optimality operator optimizes a unique robust MDP whose ambiguity sets are SA-rectangular and value function dependent. Notice that for any state s , action a and value function v , the VaR ambiguity set is a half-space { p s,a ∈ ∆ S : p T s,a v ≥ VaR α [ ˜ p T s,a v ]} dependent on the value function v . In contrast, BCR ambiguity sets are independent of any policy or value function and are constructed such that they provide high-confidence guarantees on returns of all policies simultaneously. As a result, BCR ambiguity sets tend to be unnecessarily large.

We now compute the ratio of the asymptotic radii of BCR ambiguity sets and VaR ambiguity sets.

Theorem 4.2 (Asymptotic Radii of VaR Ambiguity Sets) . For any state s and action a , let I ( { ( p ∗ s,a ) i } S -1 i =1 ) be the Fisher information of the first S -1 transition probabilities, i.e., { ( p ∗ s,a ) i } S -1 i =1 .

Define I ′ ( p ∗ s,a )=   I ( { ( p ∗ s,a ) i } S -1 i =1 ) 0 0 T 0   . Suppose that the normality assumptions on the posterior distribution ˜ P in Section 2 are satisfied. Then,

Theorem 4.2 shows that the asymptotic form of the VaR ambiguity set is an ellipsoid. Note that, in contrast to the finite-sample VaR ambiguity set P VaR , v s,a in problem (7), the asymptotic VaR ambiguity set P VaR s,a is independent of the value function v . This is because VaR α [ ˜ p T s,a v ] is convex in the value function v [39]. Therefore, the asymptotic ambiguity set P VaR s,a is simply the intersection of closed half-spaces in P VaR , v s,a , computed over all value functions v ∈ R S [9]. It is also worth noting that the radius of the asymptotic VaR ambiguity set P VaR s,a is constant. In contrast, the asymptotic radius of the BCR ambiguity sets grows with the number of states in at least one direction, as we show in the following proposition.

Theorem 4.3 (Asymptotic Radius of Bayesian Credible Regions) . For any state s and action a , let P BCR α s,a represent any Bayesian credible region. Let ξ < √ χ 2 S -1 , 1 -α / Φ -1 (1 -α ) . Suppose that the normality assumptions on the posterior distribution of ˜ P in Section 2 are satisfied. Then,

We note that Theorem 4.3 is an adaptation of Theorem 10 in [21] in RL which proves that there exist directions in which the Bayesian credible regions is at least ξ = √ χ 2 S -1 , 1 -α / Φ -1 (1 -α ) larger than VaR ambiguity sets. Since the value of ξ only grows with the number of states, we conclude that BCR ambiguity sets are sub-optimal for optimizing the percentile criterion. Figure 1a shows the growth in the ratio of radius of BCR to VaR ambiguity sets with an increasing number of states.

5 Experiments

We now empirically analyze the robustness of the VaR framework in three different domains.

Riverswim: The Riverswim MDP [46] consists of five states and two actions. The state represents the coordinates of the swimmer in the river and action represents the direction of the swim. The task of the agent is to learn a policy that would take the swimmer to the other end of the river.

Population Growth Model: The Population Growth MDP [25] models the population growth of pests and consists of 50 states and 5 actions. The states represent the pest population and actions represent the pest control measures. In our experiments, we use two different instantiations of the Population Growth Model: Population-Small and Population, which vary in the number of posterior samples.

Inventory Management: The Inventory Management MDP [56] models the classical inventory management problem and consists of 30 states and 30 actions. States represent the inventory level and actions represent the inventory to be purchased. The sale price s , holding cost c and purchase costs P are 3.99, 0.03, and 2.219. The demand is normally distributed with mean= s / 4 and standard deviation s / 6 .

Implementation details: For each domain in our experiments, we sample a dataset D consisting of n tuples of the form { s, a, r, s ′ } , corresponding to the state s , the action taken a , the reward r and the next state s ′ . We construct a posterior distribution over the models using D , assuming Dirichlet priors over the model parameters. Using MCMC sampling, we construct two datasets D 1 and D 2 containing M and K transition probability models, respectively.

We construct L train datasets by randomly sampling 80% of the models from D 1 each time. We use D 2 as our test dataset. For any confidence level δ , we train one RL agent per train dataset and method.

For evaluation, we consider two instances of the VaR framework: one (denoted by VaRN ) that assumes that ˜ P is a multivariate normal, and another (denoted by VaR ) that does not assume any structure over ˜ P . We use seven baseline methods for evaluating the robustness of our framework. They are: Naive Hoeffding [35], Optimized Hoeffding (Opt Hoeffding) [3], Soft-Robust [5], and BCR Robust MDPs with weighted ℓ 1 ambiguity sets ( WBCR ℓ 1 ) [3], weighted ℓ ∞ ambiguity sets

( WBCR ℓ ∞ ) [3], unweighted ℓ 1 ambiguity sets ( BCR ℓ 1 ) [40] and unweighted ℓ ∞ ambiguity sets ( BCR ℓ 1 ) ambiguity [40] sets. See Appendix C in the appendix for more details.

We report the 95% confidence interval of the robust performance ( δ -percentile of expected returns) of the VaR framework on the test dataset with that of other baselines for different values of δ .

Experimental Results Table 1 summarizes the performance of the VaR framework and the baselines for confidence level δ = 0 . 05 (Table 2 and Table 3 in the appendix summarizes the results for δ = 0 . 15 and δ = 0 . 3 respectively.). We observe that for confidence level δ = 0 . 05 , the VaR framework outperforms the baseline methods in terms of mean robust performance in most domains. On the other hand, for δ = 0 . 15 , the VaR framework outperforms baselines in the Population and Population-Small domains and has comparable performance to the Soft-Robust method in the Inventory domain. However, at δ = 0 . 3 we observe that the VaR framework has lower robust performance relative to the the Soft-Robust method in Riverswim and Population domains. We conjecture that this is because the Soft-Robust method optimizes the policy to maximize the mean of expected returns and is therefore able to perform better in cases where a lower level of robustness is required. However, we note that in contrast to our method, the Soft-Robust method does not provide probabilistic guarantees against worst-case scenarios.

Furthermore, as expected, we find that in many cases, the robust performance of BCR Robust MDPs with span-optimized (weighted) ambiguity sets ( WBCR ℓ 1 , WBCR ℓ ∞ ) is relatively higher than the robust performance of Robust MDPs with unweighted BCR ambiguity sets ( BCR ℓ 1 , BCR ℓ ∞ ). However, we find that even Robust MDPs with span-optimized BCR ambiguity sets are generally unable to outperform the robust performance of our VaR α framework.

Figure 2 compares the robust performance of the VaR framework and the baselines on both train and test models. The trends in the robust performance of the VaR framework and the baselines are similar on both train and test models.

6 Conclusion and Future Work

The main limitation of the VaR framework is that it does not consider the correlations in the uncertainty of transition probabilities across states and actions [19, 29, 30]. However, due to the non-convex nature of the percentile-criterion [3, 40], constructing a tractable VaR α Bellman operator that considers these correlations is not feasible. One plausible solution is to use a Conditional Value at Risk Bellman operator [11, 27] which is convex and lower bounds the Value at Risk measure. We leave the analysis of this approach for future work. Empirical analysis of the VaR α framework in domains with continuous state-action spaces is also an interesting avenue for future work.

In conclusion, we propose a novel dynamic programming algorithm that optimizes a tight lowerbound approximation on the percentile criterion without explicitly constructing ambiguity sets. We theoretically show that our algorithm implicitly constructs tight ambiguity sets whose asymptotic radius is smaller than that of any Bayesian credible region, and therefore, computes less conservative policies with the same confidence guarantees on returns. We also derive finite-sample and asymptotic bounds on the performance loss due to our approximation. Finally, our experimental results demonstrate the efficacy of our method in several domains.

Acknowledgments

The work in the paper was supported, in part, by NSF grants 2144601 and 1815275.

References

Praveen Agarwal, Mohamed Jleli, and Bessem Samet. Banach contraction principle and applications , pages 1-23. Springer Singapore, Singapore, 2018.

Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520, 2021.

Bahram Behzadian, Reazul Hasan Russel, Marek Petrik, and Chin Pang Ho. Optimizing percentile criterion using robust MDPs. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1009-1017, 2021.

Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for L ∞ -constrained s-rectangular robust MDPs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Aharon Ben-Tal, Dimitris Bertsimas, and David B. Brown. A soft robust model for optimization under ambiguity. Operations Research , 58(4):1220-1234, 2010.

Dimitri P. Bertsekas. Abstract dynamic programming . Athena Scientific, 2013. ISBN 978-1886529-42-7.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence . Oxford University Press, 2013.

G. P. Box and G. C. Tiao. Bayesian inference in statistical analysis. International Statistical Review , 43:242, 1973.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Peter Buchholz and Dimitri Scheftelowitsch. Computation of weighted sums of rewards for concurrent MDPs. Mathematical Methods of Operations Research , 89, 2019.

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in MDPs. In International Conference on Neural Information Processing Systems , pages 3509-3517. MIT Press, 2014.

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in mdps. Neural Information Processing Systems NIPS'14, page 3509-3517, Cambridge, MA, USA, 2014. MIT Press.

E. Delage and S. Mannor. Percentile optimization for Markov decision processes with parameter uncertainty. Operations Research , 58(1):203-213, 2010.

Esther Derman, Daniel Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. In Annual Conference on Uncertainty in Artificial Intelligence , 2018.

Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. A Bayesian approach to robust reinforcement learning. In Annual Conference on Uncertainty in Artificial Intelligence (UAI) , 2019.

Javier García, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(42):1437-1480, 2015.

Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesian Data Analysis . Chapman and Hall/CRC, 2014.

Charles Geyer and Glen Meeden. Asymptotics for constrained dirichlet distributions. Bayesian Analysis , 8:89-110, 03 2013.

Vineet Goyal and Julien Grand-Clement. Robust Markov decision process: Beyond rectangularity. Mathematics of Operations Research , 48(1):203-226, 2023.

Julien Grand-Clement and Christian Kroer. First-order methods for Wasserstein distributionally robust MDP. In International Conference on Machine Learning (ICML) , pages 2010-2019, 2021.

Vishal Gupta. Near-optimal Bayesian ambiguity sets for distributionally robust optimization. Management Science , 56(9), 2019.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 1861-1870, 2018.

C. A. R. Hoare. Algorithm 65: Find. Communications of the ACM , 4(7):321-322, 1961.

Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2): 257-280, 2005.

Marc Kery and Michael Schaub. Bayesian Population Analysis Using WinBUGS . Elsevier, 2012.

Sasche Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning . Springer, 2012.

Elita A. Lobo, Mohammad Ghavamzadeh, and Marek Petrik. Soft-robust algorithms for batch reinforcement learning, 2021.

Daniel J. Mankowitz, Nir Levine, Rae Ung Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, and Martin A. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In Proceedings of the 2020 International Conference on Learning Representations (ICLR) , 2020.

Shie Mannor, Ofir Mebel, and Huan Xu. Lightning does not strike twice: Robust MDPs with coupled uncertainty. In Proceedings of the 29th International Conference on Machine Learning (ICML) , pages 451-458, 2012.

Shie Mannor, Ofir Mebel, and Huan Xu. Robust MDPs with K-rectangular uncertainty. Mathematics of Operations Research , 41(4):1484-1509, 2016.

P. Massart. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. The Annals of Probability , 18:1269 - 1283, 1990.

Ariel Neufeld and Julian Sester. Robust q -learning algorithm for Markov decision processes under wasserstein uncertainty, 2023.

Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Marek Petrik and Ronny Luss. Interpretable Policies for Dynamic Product Recommendations. In Uncertainty in Artificial Intelligence (UAI) , 2016.

Marek Petrik, Mohammad Ghavamzadeh, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In International Conference on Neural Information Processing Systems , pages 2306-2314, 2016.

L. A. Prashanth. Policy gradients for CVaR-constrained mdps. In Peter Auer, Alexander Clark, Thomas Zeugmann, and Sandra Zilles, editors, Algorithmic Learning Theory , pages 155-169, Cham, 2014. Springer International Publishing. ISBN 978-3-319-11662-4.

Martin L. Puterman. Markov Decision Processes: Discrete stochastic dynamic programming . John Wiley and Sons, Inc., 2nd edition, 2005.

R. Tyrrell Rockafellar. Convex analysis . Princeton University Press, 1970.

R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk , 2:21-41, 2000.

Reazul Hasan Russel and Marek Petrik. Beyond confidence regions: Tight Bayesian ambiguity sets for robust MDPs. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NeurIPS) , 2019.

Sergey Sarykalin, Gaia Serraino, and Stan Uryasev. Value-at-risk vs. conditional value-at-risk in risk management and optimization , chapter 13, pages 270-294. INFORMS, 2008.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

A. Shapiro, W. Tekaya, J. P. da Costa, and M. P. Soares. Risk-neutral and risk-averse stochastic dual dynamic programming method. European Journal of Operational Research , pages 375391, 2013.

Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A CVaR optimization approach. In International Joint Conference on Computational Intelligence , 2019.

Lauren Steimle, David Kaufman, and Brian Denton. Multi-model Markov decision processes. Optimization Online , 2018.

Alexander L. Strehl and Michael L. Littman. A theoretical analysis of model-based interval estimation. In International Conference on Machine Learning (ICML) , 2005.

Xihong Su and Marek Petrik. Solving multi-model MDPs by coordinate ascent and dynamic programming,. In Uncertainty in Artificial Intelligence (UAI) , 2023.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An introduction . A Bradford Book, 2018.

Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML'12, page 1651-1658, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.

Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

A. W. van der Vaart. Asymptotic statistics . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.

Nithia Vijayan and L. A. Prashanth. A policy gradient approach for optimization of smooth risk measures. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , volume 216 of Proceedings of Machine Learning Research , pages 2168-2178. PMLR, 31 Jul-04 Aug 2023.

Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Huan Xu and Shie Mannor. The robustness-performance trade-off in Markov decision processes. In Proceedings of the 19th Annual Conference on Neural Information Processing Systems (NeurIPS) , 2006.

Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. Mathematics of Operations Research , 37(2):288-300, 2012.

Paul Herbert Zipkin. Foundations of inventory management . McGraw-Hill, Boston, 2000.

A Additional theoretical results

Definition A.1 (Translation Subvariance) . The operator T : R S → R S satisfies the translation subvariance property if for all vector v ∈ R S and scalar c , there exists γ ∈ [0 , 1) that satisfies

.

Definition A.2 (Monotonicity) . The operator T : R S → R S satisfies the monotonicity property if for all v ∈ R S and u ∈ R S such that v ⪯ u , T satisfies

Lemma A.3 (Contraction Mapping [6]) . The operator T : R S → R S is a contraction mapping if it satisfies monotonicity and translation subvariance properties, i.e., there exists γ ∈ [0 , 1) such that for all u ∈ R S , v ∈ R S , T satisfies

The proof of Lemma A.3 follows directly from Proposition 2.1.3 in [6]. We re-derive the proof for the sake of completeness.

Proof. Denote

Therefore for all s ∈ S ,

It follows that for all s ∈ S , which proves the result.

Applying T to these inequalities and using the translation subvariance (Definition A.1) and monotonicity (Definition A.2) properties, we obtain that for all s ∈ S ,

and therefore ∥ T u -T v ∥ ∞ ≤ γc , proving the stated result.

Lemma A.4. For any policy π , let ˆ v π and v π be the fixed point of the VaR policy evaluation operator T π VaR α and Bellman policy evaluation operator T π P . If the VaR Bellman policy evaluation operator T π VaR α dominates the Bellman policy evaluation operator T π P at ˆ v π , i.e., T π VaR α ˆ v π ⪯ T π P ˆ v π , then, the fixed point of the VaR Bellman evaluation operator T π VaR α dominates the fixed point of the Bellman evaluation operator T π P , i.e., ˆ v π ⪯ v π .

We note that in contrast to the Bellman policy evaluation operator T π P , the VaR Bellman policy evaluation operator T π VaR α is a function of the random variable ˜ P and is not dependent on the transition probabilities P assumed in this setting.

Using the assumption T VaR α ˆ v π ⪯ T π P ˆ v π , and from ˆ v π = T π VaR α ˆ v π and v π = T π P v π , we get by algebraic manipulations:

Proof.

Here P π is the transition probability function corresponding to policy π . Subtracting γ P π (ˆ v π -v π ) from the above inequality gives,

where I is the identity matrix. ( I -γ P π ) -1 is monotone as can be seen from its Neumann series.

B Proofs

B.1 Proof of Proposition 3.1

Proposition 3.1 (Validity) . The following properties of T VaR α hold for all value functions u , v ∈ R S .

The operator T VaR α is contraction mapping on R S : ∥T VaR α u -T VaR α v ∥ ∞ ≤ γ ∥ u -v ∥ ∞ .

The operator T VaR α is monotone: u ⪰ v ⇒T VaR α u ⪰ T VaR α v .

The equality T VaR α ˆ v = ˆ v has a unique solution.

Proof. From Lemma A.3, we know that an operator is a contraction mapping if it satisfies monotonicity and subvariance property.

In this proof, we will show that the VaR Bellman operator T π VaR α satisfies monotonicity and subvariance property, and therefore, is a contraction mapping.

We will use shorthand r s,a to denote the reward vector corresponding to state s and action a , i.e., r s,a = R ( s, a, · ) .

First, we show that T VaR α satisfies translation subvariance property. Consider any c ∈ R and state s . Then,

( a ) follows from simple algebraic manipulations, ( b ) follows from γc ¯ p T s,a 1 = γc , ( c ) follows from the translational invariance property of VaR α measure [41], and ( d ) follows the definition of the VaR Bellman operator T π VaR α .

Next, we show that T VaR α satisfies the monotonicity property.

Let u and v be any two value functions such that v ⪯ u . Consider any state s ∈ S . Then,

( a ) follows from the definition of the VaR α Bellman operator T VaR α , ( b ) follows from the fact that ∀ a ′ ∈ A , -max a ∈A VaR α [ ˜ p T s,a ( r s,a + γ v )] ≤ -VaR α [ p T s,a ′ ( r s,a ′ + γ v )] , ( c ) follows from the monotonicity property [41] of the VaR operator and u ⪰ v .

Thus, we prove that T VaR α is a γ -contraction mapping and a monotone operator. Since T VaR α is a contraction operator on a Banach space and from the Banach fixed point theorem [1], it follows that the operator T VaR α has a unique solution ˆ v , i.e., T VaR α ˆ v = ˆ v .

Given a policy π , a state s ∈ S , and transition probabilities P , let

represent the Bellman evaluation operator corresponding to transition probabilities P and the VaR α Bellman evaluation operator, respectively.

B.2 Proof of Proposition 3.2

Proposition 3.2 (Lower Bound Percentile Criterion) . For any δ ∈ (0 , 1 / 2 ) , if we set the confidence level α in the operator T π VaR α to δ / S , then for every policy π ∈ Π : Pr ˜ P [ˆ v π ⪯ ˜ v π |D ] ≥ 1 -δ .

Proof. Let α = δ / S . Recall that for any policy π , state s ∈ S , transition probabilities P , the Bellman evaluation operator T π P and the VaR Bellman evaluation operator T π VaR α are defined as

respectively.

Let ˆ v π be the fixed point of T π VaR α conditioned on observed transitions D , and let ˜ v π be a random variable that represents the fixed point of T π ˜ P for a given realization of the transition probabilities ˜ P . Then, applying Lemma A.4 to T π VaR α and T π ˜ P , we have, ˆ v π ⪯ v π implies

Using the equation (10), we can bound the probability that the VaR value function lower bounds the true value.

From the definition of VaR , we know that for any state s and action a ,

Therefore, using union bound and (11) in (12), we can write

Thus,

That is for each state s ,

B.3 Proof of Proposition 3.3

Proposition 3.3. Suppose that ˜ p s,a for any state s and action a , is a multivariate subGaussian with mean ¯ p s,a and covariance factor Σ s,a , i.e., E [ exp ( λ ( ˜ p s,a -¯ p s,a ) T w ) ] ≤ exp ( λ 2 w T Σ s,a w / 2 ) , ∀ λ ∈ R , ∀ w ∈ R S . Then, for any state s ∈ S , T VaR α satisfies

.

As a special case, when ˜ p s,a is normally distributed ˜ p s,a ∼ N ( ¯ p s,a , Σ s,a ) , then T VaR α for any state s ∈ S can be expressed as

Proof.

Equality ( a ) follows from the definition of VaR , ( b ) follows from using the fact that VaR α [ ˜ X ] = inf { t ∈ R : Pr[ ˜ X > t ] < 1 -α } and subtracting ¯ p T s,a w s,a on both sides, ( c ) follows from taking exponential on both sides, ( d ) follows from applying the upper-bound given by Chernoff bound for a sub-Gaussian distribution [7] i.e., Pr ( exp((˜ p s,a -¯ p s,a ) T w s,a ) ≤ exp( t -¯ p T s,a w s,a ) ) ≤ exp ( -( t -¯ p T s,a w s,a ) 2 2 w T s,a Σ s,a w s,a ) , ( e ) follows from subtracting 1 on both sides and then, taking ln on both sides, ( f ) follows from simple algebraic manipulations and ( g ) follows from taking the infimum of the solution interval of t.

Solving for t in step ( g ) , we get t = VaR α [ ˜ p T s,a w s,a ] = ¯ p T s,a w s,a -√ 2 ln( 1 / α ) √ w s,a Σ s,a w s,a which proves the stated result.

Next, we derive the VaR Bellman optimality update T VaR α ( v ) when ˜ p s,a ∀ s ∈ S , a ∈ A is normally distributed.

Consider the VaR Bellman optimality operator defined for any state s and value function v as

From the theory of multivariate Gaussian distributions [7], we know that, for any state s and action a , if ¯ p s,a is Gaussian distributed N ( ¯ p s,a , Σ s,a ) , then, ˜ p T s,a w s,a is also Gaussian distributed N ( ¯ p T s,a w s,a , w T s,a Σ s,a w s,a ) . To find the VaR α [ ˜ p T s,a w s,a ] for any state s and action a , it is sufficient

to find t such that Pr(˜ p T s,a w s,a > t ) = 1 -α .

The first equation follows from substracting ¯ p T s,a w s,a and dividing by √ w T s,a Σ s,a w s,a on both sides. The second equality follows from the definition of CDF of N (0 , 1) and the third equality follows from simple algebraic manipulations.

Substituting the value of t = VaR α [ ˜ p T s,a w s,a ] = ¯ p T s,a w s,a -Φ -1 (1 -α ) √ w T s,a Σ s,a w s,a in (13), we obtain the stated results.

The normal form of the VaR Bellman optimality operator T VaR α is useful to analyze the asymptotic properties of the VaR Bellman operator.

B.4 Proof of Theorem 3.4

Theorem 3.4 (Performance) . Let ˆ v be the fixed point of the VaR Bellman optimality operator T VaR α and π ∗ be the optimal policy in (1) . Let ρ ∗ = VaR δ [ ρ ( π ∗ , ˜ P )] denote the optimal percentile returns and ˆ ρ = p T 0 ˆ v denote the lower bound on the percentile returns computed using the Bellman operator T VaR α with α = δ / S . Then for each δ ∈ (0 , 1 / 2 ) :

Proof. We denote the optimal policy that optimizes the δ -percentile criterion by π ∗ , i.e., π ∗ ∈ arg max π ∈ Π VaR δ [ ρ ( π, ˜ P )] .

Recall that ˆ v ∈ R S is the fixed point of the VaR α Bellman optimality operator T VaR α , i.e., ˆ v = T VaR α ˆ v and ˆ ρ = p T 0 ˆ v is the returns of the corresponding policy. The operator T π P represents the Bellman evaluation operator for a given policy π ∈ Π and a transition probability model P . The Bellman evaluation operator T π P for any s ∈ S and value v ∈ R S is defined as

where w s,π ( s ) = r s,π ( s ) + γ · v .

It is known that the Bellman operator T π P is a γ -contraction mapping, monotone, and has a unique fixed point. We will use ˜ v π ∗ = T π ∗ ˜ P ˜ v π ∗ to represent the random unique fixed point of the Bellman operator T π ∗ ˜ P defined for a random realization of transition probabilities ˜ P . Furthermore, we will use p T 0 ˜ v π ∗ = ρ ( π ∗ , ˜ P ) to denote the random expected returns corresponding to policy π ∗ and random realization of the transition probabilities ˜ P .

Suppose that c ∈ R + upper-bounds the difference in performance of the VaR α policy ˆ π and optimal percentile-criterion policy π ∗ , i.e., ρ ∗ -ˆ ρ = c ⇐⇒ ρ ∗ = ˆ ρ -c .

From the definition of VaR and π ∗ , we know that

Since ˆ ρ + c upper bounds ρ ∗ , we can write

The above equation suggests that the error in the performance of the VaR α policy is upper-bounded by c if Pr[ ρ ( π ∗ , ˜ P ) -ˆ ρ ≤ c ] holds with at least δ probability.

To derive a lower bound on c , we proceed as follows.

We begin by showing that ∗

Next, we bound ∥ ˜ v π ∗ -ˆ v ∥ ∞ to obtain a lower-bound on c .

( a ) follows by simply adding and subtracting T π ∗ ˜ P ˆ v , ( b ) follows from the fact that ˜ v π ∗ and ˆ v are the fixed points of T π ∗ ˜ P and T VaR α respectively, ( c ) follows from applying the triangle inequality to the R.H.S., and ( d ) follows from the contraction property of a Bellman operator.

Rearranging and re-normalizing the above terms, we get

Then, combining (15) and (16), we get

( a ) follows from the definition of l ∞ norm, ( b ) follows from the definitions of T VaR α and T π ∗ ˜ P Bellman operators, and ( c ) follows from the fact that ˆ π ( s ) = arg max a ∈A VaR α [ ˜ p T s,a ˆ w s,a ] and thus, VaR α [ ˜ p T s, ˆ π ( s ) ˆ w s, ˆ π ( s ) ] ≥ VaR α [ ˜ p T s,π ∗ ( s ) ˆ w s,π ∗ ( s ) ] .

Therefore, for any ε > 0 , we can write

The first equation in the above follows from Pr [ ˜ p s,a w s,a > VaR 1 -1 -δ -ε S [ ˜ p T s,a ˆ w s,a ] ] < 1 -δ S for any state s and action a and applying union bound over all states yields Pr [ ˜ p s,a w s,a > VaR 1 -1 -δ -ε S [ ˜ p T s,a ˆ w s,a ] ∀ s ∈ S ] < 1 -δ . Thus, we get Pr [ ˜ p s,a w s,a ≤ VaR 1 -1 -δ -ε S [ ˜ p T s,a ˆ w s,a ] , ∀ s ∈ S ] ≥ δ . The second equation follows by simply replacing π ∗ ( s ) with the worst-case action, i.e., action that maximizes the upper bound. Comparing (14) and (18), we get

Note that VaR of returns is upper semicontinuous and thus, this implies that the infimum in the above equation is achieved at ε = 0 .

Therefore, we can write

B.5 Proof of Theorem 3.5

Theorem 3.5 (Asymptotic Performance) . Suppose that the normality assumptions on the posterior distribution ˜ P in Section 2 are satisfied. For any δ ∈ (0 , 1 / 2 ) , set α = δ / S in T VaR α . For any state s and action a , let I ( p ∗ s,a ) be the Fisher information matrix corresponding to the true transition probabilities p ∗ s,a . Furthermore, let σ 2 max = max s ∈S ,a ∈A ˆ w T s,a I ( p ∗ s,a ) -1 ˆ w s,a be the maximum over state-action pairs of the asymptotic variance of the returns estimate ˜ p T s,a ˆ w s,a . Then the asymptotic performance of the VaR framework ˆ ρ w.r.t. the optimal percentile returns ρ ∗ satisfies

Proof. As noted in Section 2, we assume that as N → ∞ , the posterior distribution of transition probabilities at any state s and action a ( ˜ p s,a ) converges in the limit to a multivariate Gaussian distribution with mean p ∗ s,a and covariance matrix I ( p ∗ s,a ) -1 / N . Hence, we can write

We know from Proposition 3.3, that VaR α of a univariate Gaussian random variable X ∼ N ( µ, σ 2 ) can be written as VaR α [ X ] = µ + Φ -1 ( α ) σ . Therefore, applying this result to the R.H.S. of Equation (6), we get

( a ) follows from the definition of VaR α under Gaussian assumptions, ( b ) and ( c ) follow from simple algebraic manipulations, and ( d ) follows from the fact that ∀ δ ∈ (0 , 1 / 2 ) , 0 ≥ Φ -1 ( 1 -1 -δ S ) ≥ Φ -1 ( δ S ) , -Φ -1 ( δ S ) = Φ -1 (1 -δ S ) , and assuming S > 1 .

We prove the second inequality in Theorem 3.5, by leveraging the sub-Gaussian bounds for a standard normal distribution N (0 , σ 2 ) to show that ∀ α ′ ∈ (0 , 1 / 2 ) , Φ -1 (1 -α ′ ) ≤ √ 2 ln( 1 / α ′ ) .

We know that for a standard normal distribution X ∼ N (0 , σ 2 ) where σ ∈ R , the following sub-Gaussian bounds holds [7]:

By definition, for a standard normal distribution N (0 , σ 2 ) , it holds

Comparing equation (20) and (21), we get

Using the result in (22) in equation (19), proves the second inequality of the theorem.

B.6 Proof of Proposition 3.6

Proposition 3.6 (Empirical Error Bound) . For any state s , action a and value function v , let ̂ VaR α [ ˜ p T s,a w s,a ] represent the empirical estimate of α -percentile of returns VaR α [ ˜ p T s,a w s,a ] and Φ f represent the cumulative density function (CDF) of the random estimate of returns ˜ p T s,a w s,a . Suppose that Φ f is differentiable at the point VaR α [ ˜ p T s,a w s,a ] and let η = Φ ′ f ( VaR α [ ˜ p T s,a w s,a ] ) represents the density of estimate of returns at point VaR α [ ˜ p T s,a w s,a ] . Let M ∗ be the number of posterior samples required to obtain empirical error ε ∈ R , with confidence 1 -ζ , where 0 < ζ < 1 , i.e., Pr [ | ̂ VaR α [ ˜ p T s,a w s,a ] -VaR α [ ˜ p T s,a w s,a ] | > ε ] ≤ ζ . Then, lim ε → 0 M ∗ ε 2 = ln( 2 / ζ ) / 2 η 2 .

Proof. To prove this theorem, we first compute the derivative of the inverse of the CDF ∂ Φ -1 f ( α ) / ∂α as follows. From the definition of the cdf Φ f and VaR , we know that, for any α ∈ (0 , 1 / 2 ) , Φ -1 f ( α ) = VaR α [ ˜ p T s,a w s,a ] .

From the inverse-function theorem, we get,

Equipped with the above result, we can now proceed to prove the main result.

To prove the result, we need to find M ∗ such that

Equation ( a ) follows from applying a first order Taylor expansion to Φ -1 f around the point α . We apply the following results to obtain a bound on M ∗ .

Let ˆ F and F represent the empirical CDF and the true CDF of a random variable ˜ Z . Suppose that the empirical estimate of the CDF ˆ F is estimated using M ∗ samples from the true distribution of ˜ Z and

0 < ζ < 1 represents the desired level of confidence guarantees, Then, from DWK inequality [31], we know that

The above equation implies that

where z t = u t = √ ln( 2 / ζ ) / 2 M ∗ .

Thus, applying equation (24) to (23), i.e., z t = √ ln( 2 / ζ ) / 2 M ∗ = εη gives M ∗ = ln( 2 / ζ ) / 2 ε 2 η 2 .

B.7 Proof of Proposition 3.7

Proposition 3.7 (Value Iteration Error) . Define the empirical VaR Bellman optimality operator T ̂ VaR α for any value v ∈ R S and state s ∈ S as ( T ̂ VaR α v ) s = max a ∈A ̂ VaR α [ ˜ p T s,a w s,a ] . Let ˆ v ∈ R S and ˆ u ∈ R S represent the fixed points of T VaR α and T ̂ VaR α respectively. Suppose that Algorithm 3.1 returns policy π k and value function u k and ∥ ˆ u -u 0 ∥ ∞ ≤ r max / 1 -γ . Then,

Furthermore, the gap between the empirical and true value function is bounded by

Proof. We begin by noting that similar to the VaR Bellman operator T VaR α , the empirical VaR Bellman operator T ̂ VaR α is also a contraction mapping because of the monotonicity of the VaR α operator and since the empirical VaR α in the algorithm is always computed from a fixed set of transition probabilities sampled from the posterior of ˜ P .

We can bound the error between the value function u k returned by Algorithm 3.1 and the fixed point ( ˆ u ) of T ̂ VaR α as

( a ) follows from applying the triangle inequality to the l ∞ norm, ( b ) follows from the fact that ∥ u k -1 -u k ∥ ∞ ≤ ε (1 -γ ) / γ and ∥ u k -1 -ˆ u ∥ ∞ ≤ 1 γ ∥ u k -ˆ u ∥ ∞ due to the contraction property of T ̂ VaR α operator, and finally, ( c ) and ( d ) follow from simply arranging the terms on both sides.

Next, we bound the approximation error ∥ u k -u π k ∥ ∞ . From the above result, we can write

Wewill use the shorthand π , T and T π k to represent the policy π k , empirical VaR Bellman optimality T ̂ VaR α , and VaR Bellman policy evaluation operator T π ̂ VaR α respectively. Suppose that δ = ˆ u -u π .

( a ) follows from the fact that ˆ u is the fixed point of T ˆ u and u π is the fixed point of T π , ( b ) follows from (25), ( c ) follows from the γ -contraction property of T operator, ( d ) follows from the definition of policy π = π k , ( e ) follows from (25), ( f ) follows from simple algebraic manipulations, ( g ) follows from taking l ∞ -norm on both sides, ( h ) follows from applying triangle inequality, and ( i ) and ( j ) follow from simple algebraic manipulations.

Next we bound the error between the optimal value function ˆ v and the fixed point of the empirical VaR Bellman optimality operator ˆ u .

( a ) follows by simply adding and subtracting T VaR α ˆ u , ( b ) follows from simply arranging the terms, ( c ) follows from applying triangle inequality, and ( d ) follows from the contraction property of the Bellman operator T VaR α .

Using simple algebraic manipulations and rearranging the terms in the above equation, we can write

Similarly, we can bound ∥ ˆ v -ˆ u ∥ ∞ in the following manner.

( a ) follows by simply adding and subtracting T ̂ VaR α ˆ v , ( b )

follows from simply arranging the terms,

( c ) follows from applying triangle inequality, and ( d ) follows from the contraction property of the Bellman operator T ̂ VaR α .

Using simple algebraic manipulations and rearranging the terms in the above equation, we can write

B.8 Proof of Proposition 3.8

Proposition 3.8 (Time Complexity) . Let r max = max s,s ′ ∈S ,a ∈A | R ( s, a, s ′ ) | . Then, Algorithm 3.1 terminates in k = ⌈ log 1 /γ ( rmax ε (1 -γ ) ) ⌉ iterations with time complexity O ( S 2 AM log 1 /γ ( rmax ε (1 -γ ) ) ) .

Furthermore, for any failure probability ζ ∈ (0 , 1) , suppose that the CDF ( Φ f ) of the random estimate of 1-step returns ˜ p T s,a ˆ w s,a is differentiable at the point VaR α [ ˜ p T s,a ˆ w s,a ] , and set η = Φ ′ (VaR α [ ˜ p T s,a ˆ w s,a ]) and M = O ( log ( S /ζ ) η 2 ε 2 (1 -γ ) 2 ) . Then with probability at least 1 -ζ , it holds that

Proof. To bound the number of iterations ( k ) required to achieve value approximation error ∥ u k -ˆ u ∥ ∞ ≤ ε , we first bound the value approximation error ∥ u k -ˆ u ∥ ∞ as follows.

( a ) follows from the definition of the VaR Bellman operator T VaR α , ( b ) follows from the contraction property of the empirical VaR Bellman operator T ̂ VaR α , ( c ) follows from applying the same procedure as in ( a ) to step ( b ) , and ( d ) follows from unrolling ( c ) over k -2 time steps and using the fact that ∥ ˆ u -u 0 ∥ ∞ ≤ r max / 1 -γ for u 0 = 0 .

We find k such that ∥ ˆ u -u k ∥ ∞ ≤ ε ,

The time complexity follows from the fact that any quantile of an array of real values can be computed in linear time using the Quick Select algorithm [23], computing 1-step return requires O ( S ) operations, and a single iteration of the loop in lines 3-12 of Algorithm 3.1 computes quantile of returns SA times.

Suppose that Φ f is differentiable at the point VaR α [ ˜ p T s,a w s,a ] and let η = Φ ′ f (VaR α [ ˜ p T s,a w s,a ]) represents the density of estimate of returns at point VaR α [ ˜ p T s,a w s,a ] .

Let for any state s and action a , ˆ w s,a = r s,a + γ ˆ v . From Proposition 3.6, we know that to guarantee Pr [ | ̂ VaR α [ ˜ p T s,a ˆ w s,a ] -VaR α [ ˜ p T s,a ˆ w s,a ] | > ε ] ≤ ζ , Mshould satisfy lim ε → 0 Mε 2 = ln( 2 / ζ ) / 2 η 2 .

Furthermore, in Proposition 3.7, we showed that ∥ ˆ v -ˆ u ∥ ∞ ≤ ∥T VaR α ˆ v -T ̂ VaR α ˆ v ∥ ∞ 1 -γ .

Then,

( a ) follows from the fact that ∥ ˆ v -ˆ u ∥ ∞ ≤ ∥T VaR α ˆ v -T ̂ VaR α ˆ v ∥ ∞ 1 -γ , ( b ) follows from arranging terms, ( c ) follows from applying the union bound, and ( d ) follows from applying Proposition 3.6 and setting confidence to ζ / S .

The final time complexity results follow from the triangle inequality ∥ ˆ v -u k ∥ ∞ ≤ ∥ ˆ v -ˆ u ∥ ∞ + ∥ u k -ˆ u ∥ ∞ and combining results in Equation (30), Proposition 3.6 and the first part of Proposition 3.8.

B.9 Proof of Proposition 4.1

Proposition 4.1 (Equivalence) . The VaR Bellman optimality operator T VaR α can be expressed as

Furthermore, the optimal VaR policy ˆ π ∈ Π D solves max π ∈ Π D min P ∈P VaR , ˆ v ρ ( π, P ) , where ˆ v ∈ R S is the fixed point of the VaR Bellman operator T VaR α , i.e., ˆ v = ( T VaR α ˆ v ) .

Proof. The first part of the proposition simply follows from the definition of the VaR operator. For any state s , action a and value function v , we can write

The minimum above exists as it minimizes a linear function on a compact set. The direction ' ≥ ' in (31) follows immediately from the constraint in the construction of the set P VaR , v s,a . The direction ' ≤ ' follows by linear program duality of the minimization over p and from the fact that VaR α [ ˜ p T s,a w s,a ] ≥ min s ′ ∈S w s,a,s ′ .

Recall that the VaR Bellman optimality operator defined for each s ∈ S and v ∈ R S as

Suppose that ˆ v is the unique fixed point of T VaR α and ˆ w s,a = r s,a + γ · ˆ v is the corresponding transition value for each s ∈ S and a ∈ A . That is

Then, we define the following robust Bellman operator ˆ T : R S → R S for each s ∈ S and v ∈ R S as

We now show that ˆ v is also the fixed point of the robust Bellman operator ˆ T .

Using the equality in (31), we now get that

Therefore, ˆ v is the unique fixed point of the SA-rectangular robust Bellman operator [24] and ˆ π is greedy with respect to the optimal robust value function of ˆ v and, therefore, is an optimal policy that solves

where ˆ v ∈ R S is the fixed point of the VaR Bellman operator T VaR α , i.e., ˆ v = ( T VaR α ˆ v ) .

B.10 Proof of Theorem 4.2

Theorem 4.2 (Asymptotic Radii of VaR Ambiguity Sets) . For any state s and action a , let I ( { ( p ∗ s,a ) i } S -1 i =1 ) be the Fisher information of the first S -1 transition probabilities, i.e., { ( p ∗ s,a ) i } S -1 i =1 .

Define I ′ ( p ∗ s,a )=   I ( { ( p ∗ s,a ) i } S -1 i =1 ) 0 0 T 0   . Suppose that the normality assumptions on the posterior ˜

distribution P in Section 2 are satisfied. Then,

For any state s and action a , we define the 1-step Bellman update function f s,a : R S → R such that

As noted in (2), we assume that as N →∞ , the posterior distribution of transition probabilities for any state s and action a ( ˜ p s,a ) converges in the limit to a multivariate Gaussian distribution with mean p ∗ s,a and degenerate covariance matrix I ( p ∗ s,a ) / N . Therefore, the 1-step returns ˜ p T s,a w s,a is asymptotically a univariate Gaussian random variable, i.e., lim N →∞ √ N ( ˜ p T s,a w s,a -p ∗ s,a T w s,a ) ∼ N ( 0 , w T s,a I -1 ( p ∗ s,a ) w s,a ) . Then, we can write

where Φ -1 represents the CDF inverse of a standard normal distribution. Equation (35) follows from the analytical form of VaR of a Gaussian random variable, i.e., for any Gaussian random variable ˜ Y ∼ N ( µ, σ 2 ) with mean µ , variance σ 2 , and confidence level α , VaR α [ ˜ Y ] = µ -Φ -1 (1 -α ) σ .

Since f s,a is convex in v when ˜ p s,a is Multivariate Gaussian distributed, we can use the definition of support function of a closed convex set [9] to retrieve the unique ambiguity set P VaR s,a .

Note that, in this case the support function is f s,a and P VaR α s,a is the unknown closed convex set

where the last statement follows from the positive homogeneity and translation property of support functions.

Rearranging the terms in the above and setting z = p -p ∗ , we get

Using basic algebraic manipulations as shown in Proposition B.1, we can write the asymptotic form of ambiguity set P VaR α s,a as an ellipsoid

Proposition B.1. Consider the two ambiguity sets given in equations (36) and (37) . These two representations are equivalent.

The first and second equations follow from the definition of P VaR s,a and simple algebraic manipulations. The third equation follows by multiplying by -1 on both sides and squaring both sides.

Recall that ˜ p s,a for any state s and action a is a categorical random variable. We assumed that the prior is a Dirichlet distribution and therefore, by the property of conjugate prior, the posterior distribution of ˜ p s,a is also a Dirichlet distribution. We assumed in Section 2 that as N →∞ , ˜ p s,a is Gaussian-distributed with mean p ∗ s,a and degenerate covariance matrix I ( p ∗ s,a ) -1 / N , i.e., covariance matrix I ( p ∗ s,a ) -1 / N has S -1 independent rows and S -1 independent columns.

For any state s and action a and transition probabilities ˜ p s,a , let ˜ q s,a ∈ R S -1 represent the first S -1 elements of the random variable ˜ p s,a , i.e., ˜ q s,a = { ( ˜ p s,a ) i } S -1 i =1 . Then, ˜ q s,a is Gaussiandistributed with the mean q ∗ s,a = { ( p ∗ s,a ) i } S -1 i =1 and invertible covariance matrix I ( q ∗ s,a ) -1 / N = I ( { ( p ∗ s,a ) i } S -1 i =1 ) -1 / N .

Thus, if we sample q s,a from N ( q ∗ s,a , I ( q ∗ s,a ) -1 / N ) and q s,a ⪰ 0 and ∑ S -1 i =1 ( q s,a ) i ≤ 1 , then we can easily compute the corresponding transition model as

Equipped with the above notation, we can proceed to prove the result.

Using the definition of positive semi-definite matrices in Equation (38), we can write

Sub-matrices of positive-semidefinite matrices are also positive-semidefinite. Thus, we can write,

( a ) holds because for any A ∈ R n × n if x T Ax ⪰ 0 ∀ x ∈ R n , then A ⪰ 0 , ( b ) follows from U T MU ⪰ 0 ∀ U , M ⪰ 0 , ( c ) follows from simple algebraic manipulations, ( d ) follows from ( q s,a -q ∗ s,a ) T I ( q ∗ s,a )( q s,a -q ∗ s,a ) ⪰ 0 because I ( q ∗ s,a ) -1 ⪰ 0 , and ( e ) follows from simply rearranging all the terms in the above equation.

Therefore, we get

Thus, it follows from construction that ∥ p s,a -p ∗ s,a ∥ I ′ ( p ∗ s,a ) -1 ≤ Φ -1 (1 -α ) .

The proof for the other direction is simply the reverse of this proof and therefore we omit it.

B.11 Proof of Theorem 4.3

Theorem 4.3 (Asymptotic Radius of Bayesian Credible Regions) . For any state s and action a , let P BCR α s,a represent any Bayesian credible region. Let ξ < √ χ 2 S -1 , 1 -α / Φ -1 (1 -α ) . Suppose that the normality assumptions on the posterior distribution of ˜ P in Section 2 are satisfied. Then,

Proof. For any state s and action a and transition probabilities ˜ p s,a , let ˜ q s,a ∈ R S -1 represent the first S -1 elements of the random variable ˜ p s,a , i.e., ˜ q s,a = { ( ˜ p s,a ) i } S -1 i =1 . Then, ˜ q s,a is Gaussiandistributed with the mean q ∗ s,a = { ( p ∗ s,a ) i } S -1 i =1 and invertible covariance matrix I ( q ∗ s,a ) -1 / N = I ( { ( p ∗ s,a ) i } S -1 i =1 ) -1 / N .

Thus, if we sample q s,a from N ( q ∗ s,a , I ( q ∗ s,a ) -1 / N ) and q s,a ⪰ 0 and ∑ S -1 i =1 ( q s,a ) i ≤ 1 , then we can easily compute the corresponding transition model as

Equipped with the above notation, we will now prove this theorem by contradiction.

For any state s and action a , suppose that lim N →∞ √ N ( P BCR α s,a -p ∗ s,a ) ⊆ lim N →∞ √ Nξ ( P VaR α s,a -p ∗ s,a ) .

Since P BCR α s,a is a credible region, it satisfies that

( a ) follows from the definition of Bayesian credible regions, ( b ) follows from multiplying by √ N on both sides and taking limit N →∞ , ( c ) follows from the assumption lim N →∞ √ N ( P BCR α s,a -p ∗ s,a ) ⊆ lim N →∞ √ Nξ ( P VaR α s,a -p ∗ s,a ) , and ( d ) follows from applying results in (37) and squaring both sides.

Since ξ < √ χ 2 S -1 , 1 -α Φ -1 (1 -α ) , there exists β > 0 , such that ξ 2 (Φ -1 (1 -α )) 2 ≤ χ 2 S -1 , 1 -β -α -β . Fix such β . Then we can write

( a ) follows from choosing β > 0 such that ξ 2 (Φ -1 (1 -α )) 2 ≤ χ 2 S -1 , 1 -β -α -β , ( b ) follows from simple algebraic manipulations, c follows because of the monotonic property of CDF, and ( d ) follows because lim N →∞ √ N ( ˜ p s,a -p ∗ s,a ) ∼ N ( 0 , I ( p ∗ s,a )) , from construction, lim N →∞ ∥ √ N ( ˜ p s,a -p ∗ s,a ) ∥ 2 I ′ ( p ∗ s,a ) -1 = lim N →∞ ∥ √ N ( ˜ q s,a -q ∗ s,a ) ∥ 2 I ( q ∗ s,a ) -1 and, lim N →∞ ∥ √ N ( ˜ q s,a -q ∗ s,a ) ∥ 2 I ( q ∗ s,a ) -1 is a sum of squares of S -1 normal random variables, which in turn is a standard Chi-squared random variable with degrees of freedom S -1 . Therefore, the probability in ( d ) can be at most 1 -β -α .

C Experiments

D Scalability of the VaR Framework

We define the VaR α q-value function (Q-function) for any policy π ∈ Π D as q π : S × A → R such that for any state s and action a , q π ( s, a ) = VaR α [ p T s,a ( r s,a + γ v π )] , where v π = T π VaR α v π is the fixed point of the VaR α Bellman evaluation operator for policy π . We will use ˆ q to denote the q-value function corresponding to the optimal VaR α policy ˆ π , i.e., ˆ π ( s ) = arg max a ∈A ˆ q ( s, a ) .

Let ˆ q θ and ˆ q ¯ θ denote the parameterized Q-value and corresponding target Q-value networks with parameters θ ∈ Θ and ¯ θ ∈ Θ respectively. Here ˆ q θ and ˆ q ¯ θ may represent any function approximators with parameter space Θ .

The optimal Q-value network ( ˆ q θ ∗ ) can be learned by simply minimizing the empirical VaR α Bellman residual error J VaR α ( θ ) , i.e.,

where data D is a list state-action (s, a) tuples that is collected using the current policy on the mean model ¯ P . In this case, the target Q-value network parameters ¯ θ is periodically updated by overwriting the target q-value network parameter with the Q-value network parameters ( θ ) .

We can also using any of the existing Actor-Critic methods [22, 42] to learn the optimal VaR α policy. In this case, instead of optimizing the policy to minimize the Bellman residual error, the algorithm will have to optimize the policy to minimize the VaR α Bellman residual error J VaR α ( θ ) .

E Implementation Details

VaR

BCR

l

1

BCR

l

l

1

l

VaRN

Soft-Robust

Naive Hoeffding

Opt Hoeffding

0

100

200

300

400

500

= 0.05 percentile returns

train

test

Methods

VaR

BCR

l

1

BCR

l

l

1

l

VaRN

Soft-Robust

Naive Hoeffding

Opt Hoeffding

1.0

0.8

0.6

0.4

0.2

0.0

= 0.05 percentile returns

×10 4

train

test

Methods

E.1 Code

The code and datasets are made available at https://github.com/elitalobo/VaRFramework.git.

E.2 Machine Specifications

We concurrently ran all experiments using 120 threads on a CPU swarm cluster with 2GB memory per thread. The total computational time was ∼ 3 hours.

VaR

BCR

l

1

BCR

l

l

1

l

VaRN

Soft-Robust

Naive Hoeffding

Opt Hoeffding

0

10

20

30

40

50

60

70

80

= 0.05 percentile returns

train

test

Methods

VaR

BCR

l

1

BCR

l

l

1

l

VaRN

Soft-Robust

Naive Hoeffding

Opt Hoeffding

1.0

0.8

0.6

0.4

0.2

0.0

= 0.05 percentile returns

×10 4

train

test

Methods

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel dynamic programming algorithm for optimizing the percentile criterion in offline Reinforcement Learning (RL) without explicitly constructing any uncertainty sets. The authors introduce the Value-at-Risk (VaR) Bellman operator for optimizing the percentile criterion, and show that it is a valid contraction mapping that optimizes a tighter lower bound on the percentile criterion compared to Robust Markov Decision Processes (RMDPs) with Bayesian credible region (BCR) ambiguity sets. The paper also provides theoretical analysis and bounds on the performance loss of the proposed framework and compares the asymptotic sizes of BCR and VaR ambiguity sets. Empirical results demonstrate the efficacy of the VaR framework in several domains.\n\nSTRENGTHS:\nAs a non-expert in this field, I find the paper interesting and addressing an important problem in offline reinforcement learning by proposing a novel dynamic programming algorithm.\nThe authors provide a thorough analysis of the proposed VaR framework, including performance guarantees and comparisons with BCR ambiguity sets.\nEmpirical results show the effectiveness of the VaR framework in various domains, outperforming baselines in terms of robust performance.\n\nWEAKNESSES:\nThe clarity of this paper could indeed be improved in several aspects. For instance, the authors use \"w\" to represent the index in line 89 and also use \"w_{s,a}\" in line 139 to represent the values, which might be confusing for readers.\n\nAdditionally, the authors attempt to answer a key question in this paper: \"Are Bayesian credible regions the optimal ambiguity sets for optimizing the percentile criterion?\" However, this problem is not clearly defined, and the criterion for optimality is not explicitly explained.\n\nQUESTIONS:\nIn Table 1 of the paper, what is the criterion for highlighting a cell in each row corresponding to different domains (Riverswim, Population, Population-Small, and Inventory)? It seems that all of the item in the first row should be highlighted.\n\nWhat will the number of samples of the posterior samples of $\\tilde{P}$ affect the experimental results?\n\nLIMITATIONS:\nThe paper does not discuss the application of the VaR framework to continuous state-action spaces in RL domains, limiting the understanding of its applicability to a wider range of problems. The time complexity of a single iteration in the algorithm, as mentioned, is O(SAN), which may make it challenging to apply the method to MDPs with large or continuous state and action spaces.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel offline RL algorithm, which is a dynamic programming algorithm to optimize a tight lower-bound approximation on the percentile criterion based on Value-at-Risk. \nA solid theoretical analysis is made to demonstrate the proposed VaR value iteration algorithm implicitly constructs tight uncertainty sets that are smaller in size than any optimized Bayesian credible region (BCR).\nFurthermore, this work provides both finite-sample and asymptotic bounds on the loss within the VaR framework.\nEmpirical results show that VaR outperforms BCR Robust MDPs on several tasks.\n\nSTRENGTHS:\n1. The proposed VaR algorithm is devised to optimize a lower bound on the percentile criterion without explicitly constructing ambiguity sets compared to BCR.\n2. The paper provides comprehensive theoretical analysis on various aspects, including the Lower Bound Percentile Criterion, finite-sample and asymptotic bounds on the loss within the VaR framework, as well as the Equivalence of VaR Bellman optimality operator and a unique robust MDP with SA-rectangular ambiguity sets. These analyses offer reliable guarantees and contribute to the robustness of the proposed approach.\n\nWEAKNESSES:\n1. The VaR algorithm heavily relies on a precisely estimated transition model $\\tilde{p} _{s,a}$. However, in offline RL, the availability of a limited dataset poses a constraint on the quality of this estimation. This limitation could potentially pose challenges for the VaR algorithm, particularly in complex tasks where the accurate modeling of the transition dynamics becomes more challenging.\n2. The proposed VaR, as originally designed, focuses on MDPs with discrete state and action spaces. \nExtending the algorithm to complex environments with continuous state and action space can be really hard.\n\nQUESTIONS:\n1. As stated in *Proposition 3.1*, the confidence level $\\alpha$ is set as $\\delta / S$. However, when the state space is large, will the calculation of $\\text{VaR} _{\\alpha}[\\tilde{p} _{s,a} ^ \\top \\omega _{s,a}]$ be challenging due to the high confidence?\n2. Additionally, the small value of $\\alpha$ in *Proposition 3.2* results in a large value in the second term of the RHS. This raises concerns about the tightness of the lower bound and its suitability for optimizing the LHS.\n3. I don't think the results of VaR and VaRN on Riverswim should be bolded in Table 1 since they are the same as others. \nFurthermore, the results for Riverswim do not differentiate between the algorithms.\nIt would be worth considering increasing the number of states in Riverswim to enhance the difficulty level and provide a more comprehensive evaluation for the algorithms.\n\nLIMITATIONS:\n**Limitations** is discussed in *Section 6*.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers robust policy optimization problem, which optimizes the percentile criterion to get a policy. It constructs an uncertainty set that contains the true model with high probability, and optimizes for the worst model in the set. Previous works use Bayesian credible regions as uncertainty sets, but it was usually considered to be overly conservative. This paper proposes Value-at-Risk based dynamic programming approach that does not explicitly construct uncertainty sets, but learns with a modified backup operator. The paper shows that theoretically and empirically the proposed approach constructs much less conservative robust policy compared to Bayesian credible regions.\n\nSTRENGTHS:\n- First paper to consider VaR framework and VaR Bellman operators, which are quite novel.\n- Concrete theoretical results for proposed framework and algorithm\n- Explicitly comparing the size of ambiguity sets theoretically seems to be a novel approach\n- The algorithm itself is quite simple and practical\n\nWEAKNESSES:\n- mainly only compares with BCR theoretically and empirically, while there exists a number of other different approaches in solving Robust MDP, as far as I know.\n- I does not think showing the returns alone in the experiments are enough to prove the efficiency of proposed methods, since any RL algorithm that is independent to given $\\delta$ will give the best return. In other words, we need to be sure that the algorithms satisfies the percentile criterion. In that sense, the empirical experiments do not seem complete.\n- While this paper only focuses on methods to solve for percentile criterion and robust MDPs, it would be interesting to also compare against other safe RL or offline RL methods to see whether the algorithm works practically compared to those methods.\n\nQUESTIONS:\n- How is this approach \"practically\" differs from CVAR based approaches? Can you provide empirical results of CVAR algorithms as well?\n- Are the experiment results statistically significant?\n- Even if BCR method assumes a lot larger region resulting in overly conservative polices, isn't it OK practically if we get good trade-off between the return and the robustness? Since we will usually optimize the robustness hyper-parameter to make the algorithm work in desired way, if BCR with $\\delta=0.5$ gives same policy to VaR with $\\delta=0.05$, it seems just fine for me. If we give $\\delta=1$, the compared algorithms will all act like standard RL algorithms, and there will exist $\\delta'$ for BCR that corresponds to the performance of $VaR$ with $\\delta$.\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a type of robust offline reinforcement learning methods, percentile criterion optimization. Percentile criterion optimization optimizes the policy under the worst \\alpha-percent of the models in an uncertainty set of models, based on the data. In this paper, the authors proposed a type of value iteration algorithms, replacing the standard Bellman backup values with value at risk (VaR) with respect to the posterior distribution of underlying models. The work is built on top of the previous work on Bayesian ambiguity sets and has a less conservative value estimates compared with previous work BCR.\n\nSTRENGTHS:\n1. This work proposed a VaR based dynamic programming method to optimize the percentile criterion, and avoid explicitly construct the uncertainty set of models. The proposed dynamic programming method is more tractable computationally compared with some methods constructing the uncertainty set explicitly.\n2. The theoretical analysis in this paper shows that the proposed method can always yield a value estimates that dominate the value function produced by BCR methods, and be a lower confidence bound in the meantime.\n\nWEAKNESSES:\n1. The notations and proofs details about the results to section 4 is not very clear.\n - a) It is not clear what is the definition of $v^\\pi$. Which dynamics/models the value function is with respect to. \n - b) The top paragraph in Page 22, in the proof of Proposition 4.1, is very unclear. Please cite and state which theorem it uses, and what's the condition and results, or derive the complete proof for this particular results.\n - c) What does the notation of a set minus a vector (or multiplied by a scalar) means in Eq (10)?\n - b) How does Theorem 4.4 shows \"the asymptotic radius of the BCR ambiguity sets grows with the number of states\" and is \"larger\" (claimed in the last paragraph of introduction), without showing the asymptotic shape of the BCR ambiguity sets. Theorem 4.4 only shows there exists certain elements in $\\mathcal{P}^{BCR}_{s,a}$ that can be from a distance (depending on $|S|$) with $\\bar{p}$.\n\n2. The proposed methods and analysis are limited to the finite state-action space. \n\n3. The empirical study only includes different variants of BCR as baselines.\n\n4. This paper lacks some discussion about related work.\n - a) It lacks of discussion on more practical, robust MDP inspired methods such as [1].\n - b) The discussion of related work is limited to the robust MDP literature and especially Bayesian robust MDP. It lacks of discussion on methods based on frequentists' lower confidence bounds, which is similar to Proposition 3.2, for example pessimistic approaches with tabular settings or linear function approximation ([2] and many similar work).\n\n[1] Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief. NeurIPS 2022.\n\n[2] Is Pessimism Provably Efficient for Offline RL. ICML 2021.\n\n[3] Robust Dynamic Programming. Garud N. Iyengar\n\nQUESTIONS:\n1. Please address the questions about soundness of section 4 in weakness #1.\n2. Why it is important to consider Percentile Criterion for offline RL, given it is often very conservative?\n3. Can authors explain the relationship between the proposed methods with robust dynamic programming [3] and the optimal ambiguity set in [paper's ref 14 and 28]. My understanding is that the proposed method is an instantiation of robust dynamic programming algorithm [3] with the uncertainty set to be the optimal ambiguity set in [paper's ref 14 and 28].\n\nLIMITATIONS:\nThe author discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using percentile criterion optimization for offline and robust reinforcement learning. Instead of using the Bellman optimality operator for value iteration, this paper suggests an algorithm using the Value-at-Risk (VaR) operator to account for pessimism. The VaR operator takes the given percentile of the return among ambiguity sets rather than the maximum. The authors show theoretically and empirically that such an operator leads to tighter value estimation with a given confidence level compared to prior works using Bayesian credible regions. Experiments on several tabular MDPs show that the VaR framework performs better than prior works using Bayesian credible regions.\n\nSTRENGTHS:\n1. The paper investigates an important problem of robust RL: Bayesian credible regions can be unnecessarily large, leading to overly conservative policies. The proposed VaR operator provably improves prior methods using the Bayesian credible regions.\n\n2. The paper provides a statistical complexity analysis and a computation complexity bound for the proposed empirical VaR operator.\n\n3. The paper is generally well-written and easy to read.\n\nWEAKNESSES:\n1. The proposed method degrades to frequentist pessimism with confidence sets (e.g., [1,2]) for Gaussian cases, which means the method is essentially the same as prior frequentist methods up to constant factors when the posterior is normally distributed, or the approximation in line 8 of Algorithm 3.1 is used (i.e., *VaRN* in the experiments). From the experiments, it seems that *VaRN* performs similarly to *VaR*. It is unclear how much improvement the proposed algorithm has over prior frequentist methods, especially when the posterior is not normally distributed.\n   \n2. The experiments are conducted on small tabular MDPs, and it is unclear how to scale up the proposed method to real-world settings. While the focus of the paper is on the theory side, real-world scale experiments would greatly strengthen the paper.\n\n\n[1] Jin, Chi, et al. \"Provably efficient reinforcement learning with linear function approximation.\" Conference on Learning Theory. PMLR, 2020.\n\n[2] Jin, Ying, Zhuoran Yang, and Zhaoran Wang. \"Is pessimism provably efficient for offline rl?.\" International Conference on Machine Learning. PMLR, 2021.\n\nQUESTIONS:\n1. How does the proposed method compare to frequentist methods, especially when the posterior is not normally distributed?\n\n2. How can we scale up the proposed method to large-scale RL problems?\n\nLIMITATIONS:\nLimitations are well discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}