Title: Bayesian Risk-Averse Q-Learning with Streaming Observations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Q-learning

risk-averse reinforcement learning

off-policy learning

Bayesian risk Markov decision process

distributionally robust Markov decision process
Abstract: We consider a robust reinforcement learning problem, where a learning agent learns from a simulated training environment. To account for the model mis-specification between this training environment and the true environment due to lack of data, we adopt a formulation of Bayesian risk MDP (BRMDP) with infinite horizon, which uses Bayesian posterior to estimate the transition model and impose a risk functional to account for the model uncertainty. Observations from the real environment that is out of the agent's control arrive periodically and are utilized by the agent to update the Bayesian posterior to reduce model uncertainty. We theoretically demonstrate that BRMDP balances the trade-off between robustness and conservativeness, and we further develop a multi-stage Bayesian risk-averse Q-learning algorithm to solve BRMDP with streaming observations from real environment. The proposed algorithm learns a risk-averse yet optimal policy that depends on the availability of real-world observations. We provide a theoretical guarantee of strong convergence for the proposed algorithm.

Bayesian Risk-Averse Q-Learning with Streaming Observations

Abstract

We consider a robust reinforcement learning problem, where a learning agent learns from a simulated training environment. To account for the model misspecification between this training environment and the real environment due to lack of data, we adopt a formulation of Bayesian risk MDP (BRMDP) with infinite horizon, which uses Bayesian posterior to estimate the transition model and impose a risk measure to account for the model uncertainty. Observations from the real environment that are out of the agent's control arrive periodically and are utilized by the agent to update the Bayesian posterior to reduce model uncertainty. We theoretically demonstrate that BRMDP balances the trade-off between robustness and conservativeness, and we further develop a multi-stage Bayesian risk-averse Q-learning algorithm to solve BRMDP with streaming observations from real environment. The proposed algorithm learns a risk-averse yet optimal policy that depends on the availability of real-world observations. We provide a theoretical guarantee of strong convergence for the proposed algorithm.

1 Introduction

Markov Decision Process (MDP) is widely applied in the Reinforcement Learning (RL) community to model sequential decision-making, where the underlying transition process is a Markov Process for a given policy. The Q-learning algorithm, which was first proposed in [28], learns the optimal Q-function of the infinite-horizon MDP. The Q-function is a function in terms of the state and action pair ( s, a ) , which denotes the expected reward when action a is taken at the initial state s and optimal action is taken at subsequent stages. In a Q-learning algorithm, the decision maker updates the value of the Q-function by constantly interacting with the training environment and observing the reward and transition each time. The training environment is often the estimate of the real environment (where the policy is actually implemented) through past observed data. Implementing the policy learned from the training environment in the real environment directly can be risky, because of model misspecification, which is caused by the lack of historical data and possible environment shift. For example, in an inventory management problem where the demand follows some unknown distribution, we have limited observations of past demand realizations or only partially observed data if unfulfilled lost orders are not observed.

One common approach is to consider a distributionally robust formulation, where one assumes the embedding unknown distribution belongs to some ambiguity set. This ambiguity set is usually chosen as a neighborhood of the reference distribution (estimated from data). For example, [23] considered the distributionally robust policy learning for the contextual bandit problems, and [32, 17, 30] studied the distributionally robust policy learning for the discounted RL setting. In particular, [14] and [15] considered Q-learning methods for distributionally robust RL with a discounted reward, where [14] constructed the ambiguity set using the Kullback-Leibler (KL) divergence whereas [15] used the

Wasserstein distance; Both works designed the Q-learning algorithm by reformulating the robust Bellman equation in its dual form, and thus, transforming the optimization over the distributions to the optimization over a scalar.

While distributionally robust formulation accounts for model misspecification and provides a robust policy that has relatively good performance over all possible distributions in the ambiguity set, it is also argued to be conservative as a price for robustness, especially when the worst-case scenario is unlikely to happen in reality. This motivates us to take a more flexible risk quantification criterion instead of only considering the worst-case performance. [31, 29] proposed a Bayesian risk optimization (BRO) framework, which adopts the Bayesian posterior distribution, as opposed to an ambiguity set, to quantify distributional parameter uncertainty. With this quantification, a risk measure, which represents the user's attitude towards risk, is imposed on the objective with respect to the posterior distribution. In this paper, we take this Bayesian risk perspective and formulate the infinite-horizon Bayesian risk MDP (BRMDP) with discounted reward.

BRMDP was first proposed in [13], where they model the unknown parameter of the MDP via a Bayesian posterior distribution. The posterior distribution is absorbed as an augmented state, together with the original physical state, and is updated at each stage through the realization of the reward and transition at that stage. A risk measure is imposed on the future reward at each stage taken with respect to the posterior distribution at that stage. Before [13], using the Bayesian technique in MDP was considered in [8] to deal with parameter uncertainty, where they simply took expectation on the total reward with respect to the posterior distribution and referred to this MDP model as Bayes-adaptive MDP (BAMDP). Based on BAMDP, there is a stream of work on model-based Bayesian RL (e.g., [25, 7, 27, 18, 3, 20, 4, 16]). Other model-free Bayesian RL methods include Gaussian process temporal difference learning [9], Bayesian policy gradient methods [11], and Bayesian actor-critic methods [26]. We referred to [12] for a comprehensive overview of Bayesian RL. More recently, [22] and [19] considered risk-sensitive BAMDP by imposing the risk measure on the total reward. In addition, [5] considered a percentile criterion and formulated a second-order cone programming by assuming a Gaussian random reward. All three works mentioned above took a non-nested formulation i.e., only one risk measure is imposed on the total reward. As pointed out in [13], one benefit of the nested formulation (i.e., a risk measure is imposed at each stage) is that the resulting optimal policy is time consistent, meaning the policy solved at the initial stage remains optimal at any later stage. In contrast, the non-nested formulation does not have this property (see [21]). In this paper, we considered the nested formulation as in [13]. Notably, [13] as well as most works of BAMDP considered the planning problem over a finite horizon. The problem can be solved efficiently only when the number of horizons is small since the number of augmented states, i.e., possible posterior distributions, increases exponentially as the time horizon increases. In this paper, we formulate the infinite-horizon BRMDP whose state space only contains the physical states. As a result, this enables us to find the stationary (time-invariant) optimal policy. We develop a Q-learning algorithm to learn this stationary optimal policy, which is quite different from the dynamic programming approach taken by [13].

On a related note, risk measures are also widely applied in the literature on safe RL (see [10] for a recent survey). In safe RL, risk measures are used to ensure that the learned policy not only performs well but also avoids dangerous states or actions that can cause large one-stage costs. risk measures are typically applied to deal with the intrinsic uncertainty of the MDP, taking into account the known transition probability or reward distribution. In contrast, our work uses risk measures to account for parametric uncertainty in the transition probability, which can prevent finding the optimal policy. Moreover, in safe RL, the risk measure is usually imposed on some cost function to satisfy some safety constraint, while we impose it on the reward objective. Although both areas use risk measures, they have different goals and frameworks.

To summarize, all the pre-mentioned work on robust RL has focused on offline learning where the agent does not interact with the real environment, but only has access to data from the training environment. In contrast, our work utilizes real-world data to update the Bayesian posterior in the offline learning process to reduce the parametric uncertainty. We also differ from Bayesian online learning, where the Bayesian posterior is updated periodically, and another optimal policy is then re-solved (regardless of computational cost) and deployed. Our approach is off-policy learning, where the learning agent receives streaming real-world observations from some behavior policy, and we focus on designing a Q-learning algorithm for the varying BRMDP that possesses statistical validity rather than simply assuming optimality is obtained after each posterior update.

2 Preliminary and Problem Statement

2.1 Standard Reinforcement Learning

Consider an RL environment M c = ( S , A , P c , r, γ ) , where S is a finite state space, A is a finite action space, r : S × A × S → R is the reward function bounded by ¯ R = max s,a,s ′ | r ( s, a, s ′ ) | , P c = { p c s,a ( · ) } ( s,a ) ∈S×A is the transition probabilities, and γ is the discount factor. The standard reinforcement learning aims to find a (deterministic) optimal policy π : S → A satisfying

2.2 Bayesian Risk Markov Decision Process (BRMDP)

The true transition probabilities, P c , are usually unknown in real problems, and we need to estimate them using observations from the real world. However, as discussed before, model misspecification due to lack of data can impair the performance of learned policy when deployed in the real environment. Hence, we take a Bayesian approach to estimate the environment and impose a risk measure on the objective with respect to the Bayesian posterior to account for this model (parametric) uncertainty. With finite state and action spaces, we can impose a Dirichlet conjugate prior ϕ s,a on the transition model of each state-action ( s, a ) pair and update the Bayesian posterior once we observe a transition from state s to s ′ by taking action a . We defer the details of updating the Dirichlet posterior on ( s, a ) to the supplementary material.

The Bayesian posterior provides a quantification of uncertainty about the transition model, which a risk-sensitive decision maker seeks to address by making robust decisions using the current model estimate. This can be done by imposing a risk measure on the future reward at each stage, which maps a random variable to a real value and reflects different attitudes toward risk. Given a Dirichlet posterior ϕ = { ϕ s,a } s,a and a risk measure ρ ξ ( f ( ξ )) that maps a function f of the random vector ξ to a real value, the value function of BRMDP under a policy π is defined as

We assume the risk measure ρ satisfies the following assumption.

Assumption 2.3. Let ξ ∈ P ( R n ) denote a random vector taking values in R n , and let f i : R n → R , i = 1 , 2 denote two measurable functions. The risk measure satisfies the following conditions:

risk measures satisfying Assumption 2.3 are similar to the coherent risk measures (see [1]), except that the sub-additivity is not required. We relax the assumption of coherent risk measure to include the risk measure VaR, which does not admit the sub-additivity.

Notice (1) takes a nested formulation, where the risk measure is imposed on the future reward at each stage as opposed to the non-nested formulation where the risk measure is only imposed once on the total reward, i.e., the value function of the classic MDP. One advantage of the nested formulation is that (1) can be expressed in a recursive form:

This enables us to define the Bellman operator to find the time-invariant optimal policy. Let π ∗ denote the optimal policy such that V ϕ, ∗ := V ϕ,π ∗ satisfying

where Π contains all randomized (and deterministic) policies. Let L ϕ be the Bellman operator such that

The following theorem ensures that L ϕ is a contraction mapping and BRMDP admits an optimal value function which is the unique fixed point of L ϕ . Its proof is in the supplementary material.

Theorem 2.4. BRMDPs possess the following properties:

L ϕ is a contraction mapping with ||L ϕ V -L ϕ U || ∞ ≤ γ || V -U || ∞ , where || · || ∞ is the sup norm in R |S| .

There exists a unique V ϕ, ∗ such that L ϕ V ϕ, ∗ = V ϕ, ∗ . Moreover,

2.5 BRMDP with VaR and CVaR

Among choices of risk measures that satisfy Assumption 2.3, we adopt two of the most commonly used risk measures, VaR and CVaR. For a random variable X properly defined on some probability space, VaR α is the α -quantitle of X , VaR α ( X ) = inf { z | P ( X ≤ z ) ≥ α } , and CVaR α normally is defined as the mean of α -tail distribution of X . However, since we are imposing the risk measure on the reward (regarded as the negative of loss) rather than the loss, we define CVaR α ( X ) = 1 α ∫ VaR α ( X ) -∞ x P (d x ) , which is the conditional expectation for X ≤ VaR α .

Before discussing solving the BRMDP, a natural question is how this risk-averse formulation is related to the original risk-neutral objective. Let V c,π denote the value function of policy π under the true MDP M c . the difference between V ϕ,π and V c,π is bounded by the following theorem.

Theorem 2.6. Suppose the Bayesian prior ϕ 0 s,a ( s ′ ) = 1 , ∀ s, a, s ′ and ρ is either VaR α or CVaR α . Let π be a deterministic policy. Let ¯ O = min s ∈S O s,π ( s ) and O s,a be the number of observed transitions from s with action π ( s ) in the dataset. Assuming O s,a > 0 for all ( s, a ) , then with probability at least 1 -¯ O -1 3 √ |S| α ,

In particular, let O = min s ∈S ,a ∈A O s,a , then with probability at least 1 -O -1 3 √ |S| α ,

where V c, ∗ is the optimal value function for M c .

Theorem 2.6 implies our reformulated BRMDP coincides with the risk-neutral MDP as more observations are available. In particular, the discrepancy is characterized in terms of the minimal number of observations for any state-action pair with the order at least O -1 3 . This indicates the BRMDP automatically balances the trade-off between robustness and conservativeness with varying numbers of observations. We defer the proof to the supplementary material due to the space limit.

2.7 Q-Learning for BRMDP

Section 2.2 ensures BRMDP is well-formulated, that is, it admits an optimal value function which can be found as the fixed point of its Bellman operator L ϕ . This allows us to derive a Q-Learning algorithm to learn the optimal policy and value function of BRMDP. Recall that an optimal Q-function is defined as

Let T ϕ denote the Bellman operator for the Q-function. That is,

The optimal Q-function Q ϕ, ∗ ( s, a ) then satisfies

where λ ∈ (0 , 1) . In a Q-learning algorithm, given a sequence of learning rates { λ t } t and an estimator of the Bellman operator ̂ T ϕ , we have the Q-learning update rule

2.8 Bayesian Risk-Averse Q-Learning with Streaming Data

Notice that in Section 2.2 and 2.7, the posterior distribution ϕ is fixed across stages. As in many previous works, the embedding model is estimated with a fixed set of past observations before solving the problem. However, this one-time estimation of the transition model does not take into consideration utilizing the new data that arrive later, which helps reduce the model uncertainty as well as the conservativeness caused by risk measure, as indicated by Theorem 2.6. This motivates us to consider a data-driven framework where we dynamically update the estimate of the model. For this purpose, we consider a multi-stage Bayesian Q-learning algorithm.

Suppose at the beginning of stage t , a batch of observations in the form of three-tuple ( s i , a i , s ′ i ) with batch size n ( t ) is available. The observation can be regarded as generated by some policy that is actually deployed in the real world, which does not depend on the learning process. The decision maker incorporates these new data to update the posterior ϕ t , with which we obtain a BRMDP model M t . We then cary out m ( t ) steps of the Q-learning update, where the initial Q-function in stage t is inherited from the previous stage t -1 . This framework is shown in Figure 1.

Observed Data

Update posterior

Update Q-function for

steps:

stage 1

Observed Data

Update posterior

Update Q-function for

steps:

stage 2

Observed Data

Update posterior

Update Q-function for

steps:

stage 3

3 Estimator for Bellman Operator

In Figure 1, a key step is to design a proper estimator ̂ T ϕ for the Bellman operator to ensure the convergence of the Q-function. In most of the existing literature on Q-learning, convergence relies on an unbiased estimator of the Bellman operator. While this is usually easy to obtain with the expectation operator which is linear, in BRMDP unbiased estimator for the Bellman operator is difficult if not impossible to obtain, because of (i) the non-linearity of risk measure (VaR and CVaR) and (ii) varying posterior distributions. Unbiased estimators for nonlinear functionals have been studied in [2]; however, their method cannot be directly applied here since the variance of the estimator is uncontrollable in the existence of varying posteriors. Instead, we use the Monto Carlo simulation to obtain an estimator with almost surely diminishing bias. We show in Theorem 4.3 and 4.4 that the Monto Carlo estimator is sufficient to guarantee the convergence of the Q-function.

3.1 Monto Carlo Estimator for VaR and CVaR with Varying Sample Size

Denote by

Given a posterior ϕ and a Q-function Q , we want to estimate

where ρ is either VaR α or CVaR α and α ∈ (0 , 1) denotes the risk level. A Monto Carlo estimator for VaR α ( f ( p | s, a, Q )) and CVaR α ( f ( p | s, a, Q )) with sample size N can be obtained by first drawing independent and identically distributed (i.i.d.) samples p 1 , p 2 , . . . , p N ∼ ϕ s,a . Denote by X i := f ( p i | s, a, Q ) , i = 1 , 2 , . . . , N . Then

where X k : N is the k th order statistic out of N samples. Both estimators are biased with a constant sample size. Increasing the sample size reduces the bias but can be computationally inefficient. As the posterior distribution concentrates more on the true model, even estimators with small sample sizes can have reduced bias. We use a varying sample size that adapts to the updated posterior distribution. Initially, we set N s,a as a fixed minimal sample size N , and decrease or increase N s,a by 1 at the beginning of each stage based on whether a new observation improves or not the estimate for the transition model on ( s, a ) . The multi-stage Bayesian Risk-averse Q-Learning algorithms with VaR (BRQL-VaR) and CVaR (BRQL-CVaR) are presented in Algorithm 1.

4 Convergence Analysis

In this section, we give the theoretical guarantee of our proposed multi-stage Bayesian risk-averse Q-learning algorithm, which ensures the asymptotic convergence of the learned Q-function to the 'optimal" Q-function. We define the random observations from the real environment and all algorithmrelated random quantities with respect to a probability space (Ω , F , P ) .

Recall that the real-world transition observations are assumed to be obtained from some behavior policy that is not controlled by the agents. This may possibly bring a definition problem of 'optimal" Q-function. Indeed, if all streaming data are generated by some greedy policy, then the number of transition observations from some state-action pair can remain small, in which case the Bayesian posterior does not converge and the learned policy is risk-averse yet random since it depends on the Bayesian posterior. As in the offline RL, where the policy is learned based on the initial fixed data set, we also define the optimal Q-function as depending on the given real-world data but differs from pure offline RL in that the given data is streaming and random.

Definition 4.1. (Data-conditional optimal Q-function)

Given an observation process ω = { ( s t,i , a t,i , s ′ t,i ) | t = 1 , 2 , . . . , i = 1 , 2 , . . . , n ( t ) } , let O ∞ s,a,s ′ ( ω ) = ∑ ∞ t =1 ∑ n ( t ) i =1 1 { ( s t,i , a t,i , s ′ t,i ) = ( s, a, s ′ ) } and O ∞ s,a ( ω ) = ( O ∞ s,a,s ′ ( ω )) s ′ ∈S denote the number of transition observations under ω . For each ( s, a ) ∈ S × A , define the limiting posterior as

where δ p c s,a ( · ) is the Dirac measure centered at true transition probability p c s,a = ( p c s,a ( s ′ )) s ′ ∈S . Q ω, ∗ is the optimal solution to BRMDP with 'posterior" ϕ ω , which satisfies Q ω, ∗ = T ϕ ω Q ω, ∗ . The data-optimal Q-function Q ω, ∗ is a random vector since the observation process is random.

With this data-conditional optimal criterion, we can now prove the convergence of the multi-stage Bayesian risk-averse Q-learning algorithm. A list of notations for different Q-functions can be found in Table 1 of the supplementary material.

Algorithm 1 Multi-stage Bayesian risk-averse Q-learning

Input: State space S , action space A , reward function r , termination stage T , Q-learning update step size { m ( t ) } T t =1 , learning rate { λ ℓ } ℓ , prior distribution { ϕ 0 s,a } s ∈S ,a ∈A , minimal sample size N , risk measure ρ ∈ {VaR α , CVaR α } with risk level α .

To prove the convergence of Q t , we prove (i) the convergence of Q ϕ t , ∗ which depends on the convergence of the posterior distribution, and (ii) the convergence of the estimator for the Bellman operator. We summarize two important results in Proposition 4.2 and Theorem 4.3. Due to the page limit, all the proofs are deferred to the supplementary material.

Proposition 4.2. (Convergence of Q ϕ t , ∗ ) Let Q ω, ∗ be defined as in Definition 4.1 and ϕ t be the posterior distribution obtained at stage t . Then the optimal Q-function under posterior distribution ϕ t converges to Q ω, ∗ almost surely. That is,

where || · || ∞ is the entry-wise sup norm.

Compared with Theorem 2.6 which characterizes the difference between the BRMDP and the original MDP by computing a concentration bound, Proposition 4.2 ensures the strong convergence of the BRMDP model to the data-conditional optimal Q-function. The next Theorem 4.3 ensures that the bias of the proposed estimator for the Bellman operator with varying sample sizes converges to zero uniformly in Q .

Theorem 4.3. (Diminishing bias for Bellman estimator) Denote by ( k ) the k th iteration of the Q-learning update. Let t k be the stage such that ∑ t k -1 t =1 m ( t ) < k ≤ ∑ t k t =1 m ( t ) . Let N ( k ) s,a denote the sample size in iteration k and ω t k denote all the past observations until stage t k . Denote by ̂ T ( k ) Q ( s, a ) = ̂ T ϕ t k N ( k ) s,a Q ( s, a ) as defined in (3) . Then, we have almost surely,

Theorem 4.3 provides a uniform convergence of the estimated Bellman estimator, which is crucial to prove Theorem 4.4. Notably, with both streaming data and risk measure, proof of Theorem 4.3 is

much more challenging than existing work on distributionally robust Q-learning, where randomness only comes from the learning process, which refers to the random data generated by the simulator. In our setting, we have mixed randomness coming from both the observation process and learning process, which complicates the analysis and differs from that of pure offline Q-learning. Also, the analysis is different from online Bayesian Q-learning, where the sample complexity results are usually constructed assuming an optimal policy is resolved and deployed instantly in each period. In contrast, we aim to prove the convergence of Q-learning in order to obtain the optimal policy.

Together with Proposition 4.2 and Theorem 4.3, we establish the convergence of Algorithm 1 in Theorem 4.4.

Theorem 4.4. Denote by Q t the Q-function given by Algorithm 1 at the end of stage t . Assume T = ∞ , and the learning rate { λ ℓ } ∞ ℓ =1 satisfies ∑ ∞ ℓ =1 λ ℓ = ∞ , ∑ ∞ ℓ =1 λ 2 ℓ < ∞ . Then, we have almost surely,

Corollary 4.5. Let Q t be defined as in Theorem 4.4 and O ∞ s,a be defined as in Definition 4.1. Assume T = ∞ and || O ∞ s,a || ∞ = ∞ , ∀ s ∈ S , a ∈ A almost surely. Then

where is the true optimal Q-function.

Corollary 4.5 is straightforward from Theorem 4.4, which ensures Algorithm 1 will eventually learn the true optimal Q-function if we can obtain an infinite number of observations for each state-action pair.

5 Numerical Experiments

5.1 Comparison Baselines:

BRQL-VaR: our proposed multi-stage Bayesian risk-averse Q-learning algorithm with risk measure VaR;

BRQL-CVaR: our proposed multi-stage Bayesian risk-averse Q-learning algorithm with risk measure CvaR;

BRQL-mean: the risk-neutral Bayesian Q-learning function. That is, the risk measure is the expectation taken with respect to the posterior distribution;

DRQL-KL: distributionally robust Q-learning algorithm with KL divergence (see [14]);

DRQL-Wass: distributionally robust Q-learning algorithm with Wasserstein distance (see [15]).

5.2 Testing examples

Example 1: Coin Toss. Consider we are playing the following game. Each time we will toss K coins and observe the number of coins that show heads, where the chance of each coin showing heads is unknown. After observing the number of heads in the last toss, we can make a guess about whether the next toss will have more heads or fewer heads. If our guess is right, we can get 1 dollar as the reward, otherwise, we need to pay 1 dollar. We also have the choice of not guessing, in which case we do not pay or get paid. We model this game as a discounted infinite-horizon MDP. The state s t ∈ S = { 0 , 1 , . . . , K } denotes the number of heads in t th toss. The actions space is A = {-1 , 0 , 1 } , where a = 1 corresponds to guess s t +1 > s t , a = -1 corresponds to guess s t +1 < s t , and a = 0 corresponds to not guess. Hence, the reward function is

Assume each coin shows a head with probability 0 . 4 . The number of coins K = 10 .

Example 2: Inventory Management. Suppose a warehouse manager runs a capacity system. At the beginning of period t , the manager observes the current inventory level s t and orders additional goods of the amount a t . An ordering cost of c = 1 is incurred for each unit of goods. The demand follows a truncated Poisson distribution with a mean of 3 and support { 0 , 1 , . . . , K } . Suppose the demand arrives during each period and is fulfilled at the end of the period. For each unit of fulfilled demand, we can obtain a profit of u = 5 . If a unit of demand is not fulfilled at the end of the period, the demand is lost and a penalty cost of q = 2 is incurred. If there are any remaining goods at the end of the period, the goods will be taken to the next period with a holding cost h = 1 per unit. The warehouse has a maximal capacity of K for the goods. We model this problem as a discounted infinite-horizon MDP with state space S = {-K,... , 0 , . . . , K } , where ( s t ) + = max( s t , 0) represents the inventory level at the beginning of period t and ( s t ) -= -min( s t , 0) represents the lost demand at the end of period t -1 . The action space is A s := { 0 , . . . , K -s + } . The reward is

We consider two settings: (I) the demand in each period uniformly distributes among { 0 , 1 , . . . , K } and (II) the demand depends on the current inventory level s . For the second setting, we will consider the case where observations are insufficient to estimate the transition probability for every state-action pair.

5.3 Experiment Setting

Coin toss. We consider stage-wise streaming observations with batch size n ( t ) = 1 and stage-wise Q-learning with number of steps m ( t ) = 1 . The minimal sample size to estimate the Bellman operator N = 10 . Initial observed data batch size n (0) = 10 . We set the radius of the KL ball and the Wasserstein ball to be 0 . 1 . The risk level for VaR and CVaR is set to 0 . 2 in Figure 2 and 0.4 in Figure 3.

Inventory Management. In Figure 4, We set K = 10 , T = 60 , m ( t ) = n ( t ) = 5 , n (0) = 20 , and N = 10 . The radius of the KL and the Wasserstein ball is 0 . 05 , and the risk level for VaR and CVaR is 0 . 2 . In Figure 5, we set the capacity K = 10 and the size of the historical data set n (0) = 30 . The policies are deployed in different environments, where the demand follows different truncated Poisson distributions with means ranging from 3 to 5 .

5.4 Results

In Figure 2- 4, we compare the value functions of different algorithms as the time stage increases. The value function of each algorithm is calculated by deploying the optimal policy in the real environment. Each curve shows the empirical expected performance and the strip around the curve shows the 95% confidence interval. In both examples, our proposed algorithms outperform the two distributionally robust Q-learning algorithms in both expected performance and variation as the time stage increases, since our proposed algorithms dynamically update the posterior to reduce the model uncertainty while two DRQL algorithms learns with fixed ambiguity set. Compared with the risk-neutral algorithm, BRQL-VaR and BRQL-CVaR achieve lower expected value functions but have smaller variations, which shows the robustness of our two proposed algorithms.

Moreover, in Figure 5 We test for the insufficient data setting, where we only have a set of historical data to estimate the transition model at the initial time stage. The value function is calculated as deploying the learned policy in different environments with demand following different Poisson distributions. Figure 5 indicates the two proposed algorithms achieve higher value functions than the risk-neutral algorithm in the more adversarial setting (with Poisson parameter less than 3 ), showing their robustness. They also obtain lower value functions than two DRQL algorithms in the adversarial setting and higher value functions in other settings, indicating the risk measure is more flexible compared to the worst-case criterion.

6 Conclusion and Limitation

In this paper, we propose a novel multi-stage Bayesian risk-averse Q-learning algorithm to learn the optimal policy with streaming data, by reformulating the infinite-horizon MDP with unknown transition model as an infinite-horizon BRMDP. In particular, we consider the two cases of risk

environment

4.00

3.75

Value function in true e

3.50

3.25

3.00

BRQL-mean

BRQL-CVaR

2.75

BRQL-VaR

DRQL-KL

2.50

DRQL-WaSS

0

10

20

30

40

50

60

stage

Value function in true environment

23.0

22.5

22.0

21.5

BRQL-mean

BRQL-CVaR

21.0

BRQL-VaR

DRQL-KL

DRQL-WaSS

10

20

30

40

50

stage

environment

4.00

3.75

Value function in true e

3.50

3.25

3.00

BRQL-mean

BRQL-CVaR

2.75

BRQL-VaR

DRQL-KL

2.50

DRQL-WasS

0

10

20

30

40

50

60

stage

Value function in different environments

BRQL-mean

5

BRQL-CVaR

3

BRQL-VaR

DRQL-KL

DRQL-WaSS

20

10

5

0

2.0

2.5

3.0

3.5

4.0

Poisson parameter

measures, VaR and CVaR, for which we design a Monte Carlo estimator with varying sample sizes to approximate the Bellman operator of BRMDP. We demonstrate the correctness of the BRMDP formulation by providing statistical guarantee and prove the strong asymptotic convergence of the proposed Q-Learning algorithm. The numerical results demonstrate that the proposed algorithms are efficient with streaming data and robust with limited data.

As discussed in the paper, one limitation of the current framework is that the behavior policy that generates real-world observations is assumed to be given. This is suitable for situations when it is expensive for the agent to interact with the real environment or to change the policy frequently as it may cause a large cost or system instability. An interesting future direction is to consider an online learning setting, where at each period the agent also needs to take action in the real world.

ACKNOWLEDGMENT

The authors gratefully acknowledge the support by the Air Force Office of Scientific Research under Grants FA9550-19-1-0283 and FA9550-22-1-0244 and the National Science Foundation under Grant NSF-DMS2053489.

AUTHOR BIOGRAPHIES

Yuhao Wang is a Ph.D. student at the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. He received his B.S. degree from the Department of Mathematics at Nanjing University, China, in 2021. His research interests include simulation and stochastic optimization and reinforcement learning. His email address is yuhaowang@gatech.edu and his web page is https://sites.gatech.edu/yuhaowang/ .

Enlu Zhou is a Professor at the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. She received the B.S. degree with highest honors in electrical engineering from Zhejiang University, China, in 2004, and received the Ph.D. degree in electrical engineering from the University of Maryland, College Park, in 2009. Her research interests include

simulation optimization, stochastic optimization, and stochastic control. Her email address is enlu.zhou@isye.gatech.edu , and her web page is https://www.enluzhou.gatech.edu/ .

References

Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical finance , 9(3):203-228, 1999.

Jose H Blanchet, Peter W Glynn, and Yanan Pei. Unbiased multilevel monte carlo: Stochastic optimization, steady-state simulation, quantiles, and other applications. arXiv preprint arXiv:1904.09929 , 2019.

Pablo Samuel Castro and Doina Precup. Using linear programming for bayesian exploration in markov decision processes. In IJCAI , volume 24372442, 2007.

Richard Dearden, Nir Friedman, and David Andre. Model-based bayesian exploration. arXiv preprint arXiv:1301.6690 , 2013.

Erick Delage and Shie Mannor. Percentile optimization for markov decision processes with parameter uncertainty. Operations research , 58(1):203-213, 2010.

Joseph L Doob. Application of the theory of martingales. Le calcul des probabilites et ses applications , pages 23-27, 1949.

Michael O Duff. Monte-carlo algorithms for the improvement of finite-state stochastic controllers: Application to bayes-adaptive markov decision processes. In International Workshop on Artificial Intelligence and Statistics , pages 93-97. PMLR, 2001.

Michael O'Gordon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes . University of Massachusetts Amherst, 2002.

Yaakov Engel, Shie Mannor, and Ron Meir. Bayes meets bellman: The gaussian process approach to temporal difference learning. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 154-161, 2003.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Mohammad Ghavamzadeh and Yaakov Engel. Bayesian policy gradient algorithms. Advances in neural information processing systems , 19, 2006.

Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning , 8(5-6):359-483, 2015.

Yifan Lin, Yuxuan Ren, and Enlu Zhou. Bayesian risk markov decision processes. In Advances in Neural Information Processing Systems , 2022.

Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust q -learning. In International Conference on Machine Learning , pages 13623-13643. PMLR, 2022.

Ariel Neufeld and Julian Sester. Robust q -learning algorithm for markov decision processes under wasserstein uncertainty. arXiv preprint arXiv:2210.00898 , 2022.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics , pages 9582-9602. PMLR, 2022.

Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete bayesian reinforcement learning. In Proceedings of the 23rd international conference on Machine learning , pages 697-704, 2006.

Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. Advances in Neural Information Processing Systems , 34:1142-1154, 2021.

Stephane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive pomdps. Advances in neural information processing systems , 20, 2007.

Alexander Shapiro. Interchangeability principle and dynamic equations in risk averse stochastic programming. Operations Research Letters , 45(4):377-381, 2017.

Apoorva Sharma, James Harrison, Matthew Tsao, and Marco Pavone. Robust and adaptive planning under model uncertainty. In Proceedings of the International Conference on Automated Planning and Scheduling , volume 29, pages 410-418, 2019.

Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust policy evaluation and learning in offline contextual bandits. In International Conference on Machine Learning , pages 8884-8894. PMLR, 2020.

Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning , 38(3):287-308, 2000.

Malcolm Strens. A bayesian framework for reinforcement learning. In ICML , volume 2000, pages 943-950, 2000.

Richard Stuart Sutton. Temporal credit assignment in reinforcement learning . University of Massachusetts Amherst, 1984.

Tao Wang, Daniel Lizotte, Michael Bowling, and Dale Schuurmans. Bayesian sparse sampling for on-line reward optimization. In Proceedings of the 22nd international conference on Machine learning , pages 956-963, 2005.

Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.

Di Wu, Helin Zhu, and Enlu Zhou. A bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics. SIAM Journal on Optimization , 28(2):1588-1612, 2018.

Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics , 50(6):3223-3248, 2022.

Enlu Zhou and Wei Xie. Simulation optimization when facing input uncertainty. In 2015 Winter Simulation Conference (WSC) , pages 3714-3724. IEEE, 2015.

Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3331-3339. PMLR, 2021.

A Dirichlet Posterior on State-action Pair

A Dirichlet distribution is parameterized by a count vector ϕ = ( ϕ 1 , . . . , ϕ k ) , where ϕ i ≥ 1 , such that the density of probability distribution p = ( p 1 , . . . , p k ) is defined as f ( p | ϕ ) ∝ ∏ k i =1 p ϕ i -1 i . For each ( s, a ) ∈ S × A , we impose a Dirichlet prior with parameter ϕ s,a = ( ϕ s,a ( s ′ )) s ′ ∈S on the unknown transition probability p s,a = ( p s,a ( s ′ )) s ′ ∈S . After we observe the transition ( s, a, s ′ ) for o s,a,s ′ times ∀ s ′ ∈ S , the posterior distribution of p s,a is also a Dirichlet distribution with parameter ϕ s,a + o s,a = ( ϕ s,a ( s ′ ) + o s,a,s ′ ) s ′ ∈S , where o s,a = ( o s,a,s ′ ) s ′ ∈S .

B Technical Proofs

We first introduce the following lemma that guarantees the maximum taken over randomized policies is equivalent to being taken over only deterministic policies.

Lemma B.1. (Deterministic Bellman operator)

Proof. For an arbitrary randomized policy π , an arbitrary V , denote by π ( s, a ) = P ( π ( s ) = a ) . We have

Hence,

The other direction holds naturally since a deterministic policy is also a randomized policy.

Lemma B.2. (Non-expansive risk measure) Suppose the risk measure ρ satisfies Assumption 2.3, then

where || · || ξ, ∞ is the sup norm with probability measure P .

Proof.

Similarly, we have

Combining together we complete the proof.

Proof of Theorem 2.4

Proof. proof of 1. ∀ ε > 0 , there exists a deterministic policy π , such that ∀ s ∈ S

Then,

Since ε can be chosen arbitrarily, ||L ϕ V -L ϕ U || ≤ γ || V -U || ∞ . Switching the position of V and U , we obtain the desired result.

proof of 2. For the first part, since L ϕ is a contraction mapping, there exists a unique V ϕ, ∗ such that L ϕ V ϕ, ∗ = V ϕ, ∗ . For the second part, let π ′ be an arbitrary randomized policy. We have

where the third equality holds because of Lemma B.1. Furthermore, by Assumption 2.3.2, we get

Define L ϕ π ′ such that ∀ s ∈ S and V ∈ R |S| ,

By the same induction as (4), we can derive

Define and

By Assumption 2.3 and Lemma B.2, we have

As k →∞ ,

where V ϕ,π ′ is the value function of BRMDP with posterior ϕ and policy π ′ . Then from (5) we obtain

By taking k →∞ , we have V ϕ, ∗ ( s ) ≥ V ϕ,π ′ ( s ) . Since π ′ is chosen arbitrarily, we get the desired result.

Proof of Theorem 2.6

Proof. Denote by

We first bound | f ( p | s, a, V ) -f ( p ′ | s, a, V ) | in terms of ∥ p -p ′ ∥ ∞ for ∥ V ∥ ∞ ≤ ¯ R 1 -γ . To see this,

Now consider p ∼ ϕ s,a . Since ϕ 0 s,a = 1 , we have the posterior parameter ϕ s,a ( s ′ ) = o s,a,s ′ +1 , where o ( s, a, s ′ ) is the number of transition from s to s ′ taken action a in the data set. Let ̂ M denote the empirical MDP generated with the same data which are used to estimate ϕ , with the empirical transition probability ̂ p s,a ( s ′ ) = o s,a,s ′ O s,a . Let p c s,a be the true transition distribution. Since { o s,a,s ′ } s ′ ∈S follows the multinomial distribution with parameter O s,a ; p c s,a , we have

by Chebyshev inequality. Define

and

Notice

where the second last inequality holds by Chebyshev inequality. Replacing a with π ( s ) , we obtain with probability at least 1 -α |S| O -1 3 s,π ( s ) ≥ 1 -α |S| ¯ O -1 3 ,

. This further implies with probability at least 1 -α |S| ¯ O -1 3 , P p ∼ ϕ s,π ( s ) ( ∥ p -p c s,π ( s ) ∥ ∞ ≥ 3 √ |S| α O -1 3 s,π ( s ) ) ≤ ∑ s ′ ∈S P p ∼ ϕ s,π ( s ) ( | p ( s ′ ) -p c s,π ( s ) ( s ′ ) | ≥ 3 √ |S| α O -1 3 s,π ( s ) ) ≤ α. Let U = { p |∥ p -p c s,π ( s ) ∥ ∞ ≤ 3 √ |S| α O -1 3 s,π ( s ) } . Since with probability at least 1 -α |S| ¯ O -1 3 , VaR α is the α -quantile, and P ϕ s,a ( U ) ≥ 1 -α , we have inf p ∈ U f ( p | s, π ( s ) , V ) ≤ ( VaR α ) p ∼ ϕ s,π ( s ) ( f ( p | s, π ( s ) , V )) ≤ sup p ∈ U f ( p | s, π ( s ) , V ) .

Hence with probability at least 1 -α |S| ¯ O -1 3 ,

where the second last equality is by (6). Since this holds for all s , then we obtain with probability 1 -α ¯ O -1 3 ,

holds for all s ∈ S .

Finally, let L ϕ,π denote the Bellman operator for BRMDP with posterior ϕ and policy π , that is V ϕ,π = L ϕ,π V ϕ,π = ( VaR α ) p ∼ ϕ s,π ( s ) ( f ( p | s, π ( s ) , V ϕ,π )) . Similar as Theorem 2.4, L ϕ,π is a γ -contraction mapping. We have

Since ∥ V c,π ∥ ∞ ≤ ¯ R 1 -λ , we have the second term |L ϕ,π V c,π -V c,π | ≤ 3 √ |S| α ¯ O -1 3 |S| ¯ R 1 -γ by (8). Maximize over s on both sides, we obtain

Hence,

For ρ is CVaR α , notice by (7), we have with probability at least 1 -α |S| O -1 3 s,a ,

Hence, with probability at least 1 -α |S| O -1 3 s,π ( s ) ≥ 1 -α |S| ¯ O -1 3 ,

Recall CVaR α ( X ) is the conditional expectation on { X ≤ VaR α ( X ) } . Let U = { p |∥ p -p c s,π ( s ) ∥ ∞ ≤ 3 √ |S| α O -1 3 s,π ( s ) } and W = { p | f ( p | s, π ( s ) , V ) ≤ ( VaR α ) p ∼ ϕ s,π ( s ) ( f ( p | s, π ( s ) , V )) } for ∥ V ∥ ∞ ≤ ¯ R 1 -λ . Then, with probability at least 1 -α |S| ¯ O -1 3 ,

Then, following the same proof as for VaR α , with probability at least 1 -α ¯ O 1 3 , we can bound

This completes the proof of the first part. For the second part, let π c be the optimal policy for M c . Then we have for both VaR α and CVaR α , with probability at least 1 -α ¯ O -1 3 ( π c ) ≥ 1 -αO -1 3 ,

Recall π ∗ is the optimal policy for BRMDP, then

Combining together the proof is completed.

Proof of Proposition 4.2

Proof. First notice

Since || Q ω, ∗ || ∞ ≤ ¯ R 1 -γ almost surely, we can prove the convergence of Q ϕ t , ∗ if we can show that almost surely for each ( s, a ) ∈ S × A ,

Fix a state-action pair ( s, a ) , then for any observation process ω , if || O ∞ s,a ( ω ) || ∞ < ∞ , ϕ t s,a = ϕ ω s,a after some time stage τ and (9) clearly holds for such ω . Otherwise, || O ∞ s,a ( ω ) || ∞ = ∞ . For those ω 's, by Bayesian consistency [6], we know that for any neighborhood of p c s,a with parameter ε > 0 , which is defined as U ε = { p ∈ R |S| + ∣ ∣ ∑ x ∈S p ( x ) = 1 , | p ( x ) -p c s,a ( x ) | ≤ ε, ∀ x ∈ S } , lim t →∞ P ( p ∈ U ε | ϕ t s,a ) = 1 almost surely.

For any probability mass function p ∈ R |S| + , Recall f ( p | s, a, Q ) = E s ′ ∼ p [ r ( s, a, s ′ ) + γ max b ∈A Q ( s ′ , b )] , we have ∀ p ∈ U ε ,

In the following we fix a sample path ω such that || O ∞ s,a ( ω ) || ∞ = ∞ and drop the notation of ω for simplicity. We have almost surely:

If ρ is VaR α : Since P ( p ∈ U ε | ϕ t s,a ) → 1 , for any risk level α > 0 , P ( p ∈ U ε | ϕ t s,a ) > 1 -α for t large enough. Since VaR α is the α -quantitle, we have

Similarly,

Combining the two inequalities above, we have

Since this holds for any Q with || Q || ∞ ≤ ¯ R 1 -γ and ε > 0 , we know

This completes the proof.

If ρ is CVaR α :

Since VaR ϕ t s,a α ( f ( p | s, a, Q )) → f ( p c s,a | s, a, Q ) and ϕ t s,a ( U c ε ) → 0 , we have for large enough t ,

Then we obtain

Again, by arbitrary ε and the uniformness in Q , we obtain the desired result.

Proof of Theorem 4.3

Proof. Proof sketch. The complete proof is technical. We first provide a proof sketch. Intuitively, the bias term converges to 0 since we either have posterior to concentrate on the true parameter or have an increasing sample size for the Monto Carlo estimator, both of which reduce the bias term to zero asymptotically. However, a major difficulty is uniform convergence (in terms of all possible values of Q and sample size N ), for which existing results do not give a straightforward guarantee. We prove the results by considering the two cases. First, when infinite observations are available, i.e., || O ∞ s,a || = ∞ , we construct i.i.d. samples h ( p i ) ≤ f ( p i | s, a, Q ) with the same sampled p i for an arbitrary given ε , where f ( p i | s, a, Q ) is defined as in (2). with probability at least 1 -ε , h ( p i ) takes a value no less than f ( p i | s, a, Q ) by a constant multiple of ε . with probability at least ε , h ( p i ) takes a value bounded by a constant. With the help of the Stirling formula and subtle calculation, we can bound the expectation of order statistics of h ( p i ) , which further lower bounds the Bellman operator estimator. The upper bound can be obtained in a similar way. Together we can bound the bias term. Second, in the case of limited observations, we prove a uniform convergence of empirical distribution for f ( p i | s, a, Q ) when Q belongs to a bounded set. We carefully partition the probability space as a union of disjoint rectangular sets, to which the Glivenko-Cantelli theorem can be applied and the uniform convergence of the empirical distributions follows. Combining the two cases we complete the proof.

Formal proof. Again, we fix a state-action pair ( s, a ) , an observation process ω for which we drop the notation for simplicity. Let Q be any Q-function such that || Q || ∞ ≤ ¯ R 1 -γ . We first prove for the VaR risk functional.

Estimator for VaR

we consider the two cases:

O ∞ s,a = ∞ , then ϕ t s,a is consistent at the true parameter p c s,a . ∀ ε > 0 , P ( p ∈ U ε | ϕ t s,a ) → 1 almost surely for such ω , where U ε = { p ∈ R |S| + ∣ ∣ ∑ x ∈S p ( x ) = 1 , | p ( x ) -p c s,a ( x ) | ≤ ε, ∀ x ∈ S } . Then ∀ ε > 0 , P ( p ∈

U ε | ϕ t s,a ) ≥ 1 -ε for all large t . At iteration k such that t k large enough, we obtain p ( k ) 1 , p ( k ) 2 , . . . , p ( k ) N ( k ) s,a ∼ ϕ t k s,a . For each p ( k ) i , we have P ( p ( k ) i ∈ U ε | ϕ t k s,a ) ≥ 1 -ε . Furthermore, since ϕ t k s,a is always smooth function, we can find Φ ( k ) i ∈ U ε such that P ( p ( k ) i ∈ Φ ( k ) i | ϕ t k s,a ) = 1 -ε . Recall

Define

It is easy to see f ( p ( k ) i | s, a, Q ) ≥ h ( p ( k ) i ) . Notice the distribution of sampled p ( k ) i only depends on ϕ t k s,a , N ( k ) s,a , which are further determined by ω t k . Conditioned on ω t k , h ( p ( k ) i ) , i = 1 , 2 , . . . , N ( k ) s,a are i.i.d. random variables taking two values inf p ∈ U ε f ( p | s, a, Q ) and inf p f ( p | s, a, Q ) with probability at least 1 -ε and ε , respectively. Notice inf p f ( p | s, a, Q ) ≥ -¯ R 1 -γ . And

By (10), we have | inf p ∈ U ε f ( p | s, a, Q ) -f ( p c s,a | s, a, Q ) | ≤ ε |S||| Q || ∞ 1 -γ . By (11), we have | f ( p c s,a | s, a, Q ) -VaR ϕ t k s,a α ( f ( p | s, a, Q )) | ≤ ε |S||| Q || ∞ 1 -γ for k large enough. Hence, we obtain

where || Q || ∞ again can be bounded by ¯ R 1 -γ . Denote by X i = f ( p ( k ) i | s, a, Q ) and Y i = h ( p ( k ) i ) . We first bound the conditional expectation of the order statistic Y ⌈ N ( k ) s,a α ⌉ : N ( k ) s,a , i.e., E [ Y ⌈ N ( k ) s,a α ⌉ : N ( k ) s,a | ω t k ] .

Since { Y i | ω t k } are i.i.d., we can compute the conditional mass function of Y ⌈ nα ⌉ : n as

We now lower bound the conditional expectation of Y ⌈ nα ⌉ : n by upper bound (12). First, we can show ( n j ) ε j (1 -ε ) n -j is decreasing in term of j for j > nε . To see this, take logarithm on the right hand side we have

Compute the difference

Then η ( j +1) -η ( j ) < 0 ⇐⇒ j ≥ ( n +1) ε -1 . Hence for ε < α and j ≥ nα ≥ nε > ( n +1) ε -1 , η ( j ) is decreasing. Hence we can upper bound (12) by

Then we have

For n ≥ 2 α (1 -α ) , we have

Then

where C = 4 √ (1 -α ) α 3 2 √ π is a constant and β = ε α (1 -ε ) 1 -α α α ( 1 -α 2 ) 1 -α . For ε small enough, we can ensure β < 1 . Let

Then, since log β < 0 , ζ ( x ) attains the maximum at x = -2 log β . Then we have

For n ≤ ⌊ 2 α (1 -α ) ⌋ , we have

Let where C ′ ( ε ) → 0 as ε → 0 . Then we have

Hence we have almost surely,

where ˜ C ( ε ) → 0 as ε → 0 . Similarly by constructing

We can obtain

almost surely for k large enough and ̂ C ( ε ) → 0 as ε → 0 .

Notice T ϕ t k Q ( s, a ) = VaR ϕ t k s,a α ( f ( p | s, a, Q )) . Furthermore, since both ˜ C ( ε ) and ̂ C ( ε ) do not depend on Q and by arbitrary ε , we obtain

If O ∞ s,a < ∞ , then the posterior distribution on ( s, a ) remains the same as ϕ ω s,a for all large t and the sample size N ( k ) s,a tends to infinity. In this case, by Glivenko-Cantelli theorem,

we know almost surely the empirical distribution conditioned on ϕ ω s,a (which is further determined by ω ), F ( k ) N ( k ) s,a , for the sampled p ( k ) 1 , p ( k ) 2 , . . . , p ( k ) N ( k ) s,a uniformly converges to the distribution ϕ ω s,a as sample size N ( k ) s,a goes to infinity. Denote by X ( k ) i = f ( p ( k ) i | s, a, Q ) = ∑ s ′ ∈S p ( k ) i ( s ′ )( r ( s, a, s ′ )+ γ max b ∈A Q ( s ′ , b )) = ∑ s ′ ∈S d s ′ p ( k ) i ( s ′ ) = d ⊤ p ( k ) i . We want to show uniform convergence of empirical distribution of { X ( k ) i } i =1 , 2 ,...,N ( k ) s,a conditioned on ϕ ω s,a for d ∈ D uniformly, where D contains all the possible value d can take. Denote by G d n the empirical distribution conditioned on ω t k of { X ( k ) i } i =1 , 2 ,...,n in terms of d and G d the true distribution of d ⊤ p for p ∼ ϕ ω s,a . We want to show

Notice that | d | ≤ ¯ R 1 -γ and ( d + ε 1) ⊤ ϕ = d ⊤ ϕ + ε , we have G d n ( x ) = G d + ε 1 n ( x + ε ) . Hence we only need to prove for D = { d ∈ R |S| : M 1 ≥ d ≥ 1 } , where M = 2 ¯ R 1 -γ +1 . Denote by P n ( K ) = 1 n ∑ n i =1 1 { p i ∈ K } where p 1 , · · · , p n ∼ ϕ ω s,a are i.i.d samples.

For an arbitrary ε = 1 ˜ N , we have

Since the last probability is just some probability of rectangulars, it can be expressed by finite number of cumulative distribution function, which by the Glivenko-Cantelli theorem, uniformly converge to the true distribution. Then, almosdt surely there exists n ε > 0 (independent of d, ε, k i ), for n ≥ n ε , we have each

Hence we obtain

Since this holds for any d, x , we have

The last term converges to zero as ˜ N →∞ . This is indicated by

Since each 0 ≤ q s ≤ 1 and its marginal distribution ϕ ω s,a ( q s ) is continuous, it is uniformly continuous in 0 ≤ q ≤ 1 . Hence each P q ∼ ϕ ω s,a ( x d s < q s ≤ x d s + ( |S|1) M ˜ N ) converge uniformly to 0 as ˜ N →∞ . Hence by arbitrary ˜ N , we have

Similarly, we can obtain the other side as

Hence,

We obtain the uniform convergence for empirical distribution in d , or equivalently, Q , conditioned on the observation process ω . Hence the quantile estimator X ⌈ nα ⌉ : n → VaR ϕ ω s,a α ( Q ) uniformly in Q almost surely. Furthermore, since X i is bounded,

Estimator for CVaR

Given sample p ( k ) 1 , p ( k ) 2 , . . . , p ( k ) N ( k ) s,a and X i = f ( p ( k ) i | s, a, Q ) = ∑ s ′ ∈S p ( k ) i ( s ′ )[ r ( s, a, s ′ ) + γ max b ∈A Q ( s ′ , b )] . The Monto Carlo estimator for CVaR ϕ t k s,a α ( f ( p | s, a, Q )) is 1 ⌈ N ( k ) s,a α ⌉ ∑ ⌈ N ( k ) s,a α ⌉ j =1 X j : N ( k ) s,a . Again we consider the two cases:

O ∞ s.a = ∞ . Then we know almost surely ϕ t s,a is consistent at p c s,a . Then by the proof for bias of VaR estimator, ∀ 1 > ε > 0 , E [ X ⌈ N ( k ) s,a ε ⌉ : N ( k ) s,a | ω ] → VaR ϕ t k s,a ε ( f ( p | s, a, Q )) as

k →∞ uniformly in Q almost surely. Then

Since E [ X ⌈ N ( k ) s,a ε ⌉ : N ( k ) s,a | ϕ t k s,a ] → VaR ϕ t k s,a ε ( f ( p | s, a, Q )) , VaR ϕ t k s,a ε ( f ( p | s, a, Q )) → f ( p c s,a | s, a, Q ) , CVaR ϕ t s,a α ( f ( p | s, a, Q )) → f ( p c s,a | s, a, Q ) uniformly for -¯ R 1 -γ ≤ Q ≤ ¯ R 1 -γ as k → ∞ and ε is chosen arbitrarily, we have lim inf k →∞ { E [ 1 ⌈ N ( k ) s,a α ⌉ ∑ ⌈ N ( k ) s,a α ⌉ j =1 X j : N ( k ) s,a | ω t k ] -CVaR ϕ t k s,a α ( f ( p | s, a, Q )) } ≥ 0 uniformly in Q almost surely. Similarly we can also prove lim sup k →∞ { E [ 1 ⌈ nα ⌉ ∑ ⌈ nα ⌉ j =1 X j : n | ω t k ] -CVaR ϕ t k s,a α ( f ( p | s, a, Q )) } ≤ 0 uniformly in Q almost surely. Combining the two inequalities together, we obtain

For O ∞ s,a ( ∞ ) < ∞ . By proof of VaR estimator we know the empirical distribution of f ( p ( k ) i | s, a, Q ) conditioned on ω , converge uniformly to ϕ ω s,a . Hence the CVaR estimator also has the desired result.

Lemma B.3. [24] Consider a stochastic process ( α t , ∆ t , g ) , t ≥ 0 , where α t , ∆ t , g : X → R satisfy the equations

Let P t be a sequence of increasing σ -fields such that α 0 and ∆ 0 are P 0 -measurable and α t , ∆ t and F t -1 are P t -measurable. Let || · || W denote some weighted maximum norm. Assume that the following hold:

the set X is finite.

∥ E { g ( · ) | P t }∥ W ≤ κ ∥ ∆ t ∥ W + c t , where κ ∈ [0 , 1) and c t converges to zero w.p.1.

Var { g ( x ) | P t } ≤ K (1 + ∥ ∆ t ∥ W ) 2 , where K is some constant.

Then, ∆ t converges to zero with probability one (w.p.1).

Proof for Theorem 4.4.

Proof. Following the same notation in Theorem 4.3. Denote by Q ( k ) the Q-function in iteration k . Define

Then by Theorem 4.3 and observing || Q ( k ) || < ¯ R 1 -γ almost surely, we have almost surely,

Denote by ∆ ( k ) = Q ( k ) -Q ω, ∗ . ∆ ( k ) satisfies

∆ ( k +1) ( s, a ) = (1 -λ k )∆ ( k ) + λ k F ( k ) , where F ( k ) = ̂ T ( k ) Q ( k ) -Q ω, ∗ . Let H ( k ) be the σ -field generated by {{ ∆ ( κ ) } k κ =1 , { λ ( κ ) } k κ =1 , { F ( κ ) } k -1 κ =1 } . We have | E [ F ( k ) ( s, a ) |H ( k ) ] | = | E [ E [ F ( k ) ( s, a ) | ω t k , H ( k ) ] |H ( k ) ] | = | E [ T ( k ) ( Q ( k ) )( s, a ) -Q ϕ t k , ∗ ( s, a ) + Q ϕ t k , ∗ ( s, a ) -Q ω, ∗ ( s, a ) + bias ( k ) ( s, a ) |H ( k ) ] | = | E [ ̂ T ( k ) ( Q ( k ) -Q ϕ t k , ∗ ) |H ( k ) ] + E [ Q ϕ t k , ∗ ( s, a ) -Q ω, ∗ ( s, a ) |H ( k ) ] + E [ bias ( k ) ( s, a ) |H ( k ) ] | ≤ γ E [max s ′ ∈S | max b ∈A Q ( k ) ( s ′ , b ) -max b ∈A Q ϕ t k , ∗ ( s ′ , b ) ||H ( k ) ] + E [ || Q ϕ t k , ∗ -Q ω, ∗ || ∞ |H ( k ) ] + E [ | bias ( k ) ( s, a ) ||H ( k ) ] ≤ γ E [ || Q ( k ) -Q ϕ t k , ∗ || ∞ |H ( k ) ] + E [ || Q ϕ t k , ∗ -Q ω, ∗ || ∞ |H ( k ) ] + E [ | bias ( k ) ( s, a ) ||H ( k ) ] ≤ γ E [ || Q ( k ) -Q ω, ∗ ||∞|H ( k ) ] + (1 + γ ) E [ || Q ϕ t k , ∗ -Q ω, ∗ ||∞|H ( k ) ] + E [ | bias ( k ) ( s, a ) ||H ( k ) ] ≤ γ || ∆ ( k ) || ∞ +(1 + γ ) E [ || Q ϕ t k , ∗ -Q ω, ∗ || ∞ |H ( k ) ] + E [ | bias ( k ) ( s, a ) ||H ( k ) ] .

where || · || ∞ denote the entry-wise sup norm. The second equality holds since the Monto Carlo sampling procedure in iteration k only depends on the posterior ϕ t k and sample size N ( k ) s,a , which are completely determined by the past observations ω t k . By Proposition 4.2 and Theorem 4.3, we know both || Q ϕ t k , ∗ -Q ω, ∗ || ∞ and | bias ( k ) ( s, a ) | converge to 0 almost surely. Since they are both bounded, by the bounded convergence theorem the last two terms in the last inequalities converge to zero almost surely. Furthermore, since | F ( k ) ( s, a ) | ≤ ¯ R + || ∆ ( k ) || ∞ , we have

By Lemma B.3, we obtain || ∆ ( k ) || ∞ converges to 0 almost surely, which completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a Bayesian risk-averse Q-learning algorithm to tackle the setting of Bayesian risk MDP, which uses Bayesian posterior to estimate the transition model and impose a risk functional to account for the model uncertainty. The claim is that the proposed algorithm learns a \"risk-averse yet optimal policy\", which has theoretical guarantee of strong convergence.\n\nSTRENGTHS:\nThe paper proposes an interesting formalization of an algorithm for Bayesian risks in MDP. The paper is scientifically sound and provides a few interesting results, among others with respect to the convergence (to the \"optimal\") in the context of infinite data. The paper explains why Monte-Carlo estimators are useful in practice and provides a few interesting theoretical properties. Numerical experiments provide illustrations of the theoretical analysis in the context of two relevant (small-scale) MDPs.\n\nWEAKNESSES:\nEven though related work is overall well-discussed, the paper could be more clear about the novelty of the different parts (e.g. Theorem 2.2 and Theorem 2.3 seem relatively generic and even though I'm not an expert in the BRMDP setting, I believe close theorems exist in the literature). \n\nThe notations and overall formalization might benefit from a few (minor) improvements to improve the readability (see additional comments and questions).\n\nAdditional comments:\n- In Equation 1, $d_i$ and $p_i$ do not seem to be formally defined (even though we can guess what they refer to).\n- In Equation 1, \\rho is used with a subscript that depends on a sampling from a Dirichlet posterior but \\rho was introduced line 114 without any subscript (the meaning of the subscript might not be fully obvious).\n- line 97: $r$ is not defined (even though we can guess what they refer to)\n- line 108: (s,a) is not in math mode (italic).\n- line 109: \"??\" instead of a reference to the appendix.\n\nQUESTIONS:\nMain question:\n- Theorems are mostly provided as fully original. What are the closest related theorems from the literature?\n- The inventory management problem is considered in two settings as can be read in the supplementary material. The second one is described as \"(...) the demand depends on the current inventory level s. (...) we will consider the case where observations are insufficient to estimate the transition probability for every state-action pair\". What is actually meant by that? And also, why are there two settings described in the supplementary material but only one in the main paper?\n\nAdditional questions:\n- The abstract mentions the following: \"The proposed algorithm learns a risk-averse yet optimal policy that depends on the availability of real-world observations.\" This seems unclear from the abstract because a risk averse policy will in general need make a tradeoff with the best expected return. Do you mean \"risk averse policy that converges to the optimal one in the context of unlimited data\"?\n- line 117, $\\xi$ is defined as a probability distribution over $\\mathbb R^n$, but is described as a \"random vector taking values on $\\mathbb R^n$\". Can you clarify?\n- Line 123: Could there be some intuitions for the different parts of assumption 2.1? In particular Line 124-128 gives lightweight information about how it is slightly similar to the notion of coherent risk measure but that notion and the difference with it are not detailed. Why is sub additivity not included, why is 2.1.3 important?\n\nLIMITATIONS:\nSome limitations are provided.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends previous work on Bayesian Risk-averse MDPs (BRMDPs), an informed alternative to an ambiguity set in the infinite horizon setting. In doing so, the authors first present a nested BRMDP formulation for the state value function. Then, they show that difference between optimal value functions for BRMDP and the true MDP is bounded. Afterwards, the authors presented a multi-stage risk-averse Q-learning algorithm with periodic posterior updates and a Monte Carlo estimator for the proposed risk-averse bellman operators. Simple simulations are presented to verify the better performance as well as the lesser variance of the proposed infinite horizon BRMDP formulation. It should be noted that the Bayesian posterior and Risk functionals are well-defined and that the state and action spaces discussed in the paper are both finite. The work does hold merit, and the findings could be communicated to a larger audience through a conference like NeuRIPS.\n\nSTRENGTHS:\nThe paper is generally well-written:\n\n1. The infinite horizon BRMDP formulation is novel and the accompanying recursive Bellman equations follow nicely.\n\n2. Detailed proofs of the distance bound for the State function and the convergence analysis are provided.\n\n3. Experimental results are discerned in a manner that relates directly to claims of risk aversion (smaller variances) and performance gains.\n\nWEAKNESSES:\n1. Apart from the assumption of the availability of a behavioral policy to generate real-world data, the rules for updating the sample sizes appear to be a heuristic at best. (ref. Algorithm 1)\n\n2. The Monte Carlo estimators for the Bellman operators, in addition to generating real-world samples to update the Bayesian posterior, will make the algorithm extremely slow. \n\n3. The state and action spaces under consideration appear to be finite, thus limiting the evaluation to more demanding experimental setups.\n\nQUESTIONS:\n1. In line 180, page 5, section 2.5, it is stated that a batch n(t) of observations is available. What is the assumption for the smallest batch size?\n\n2. What is the reasoning behind using the Monte Carlo estimator?\n\n3. Why is the benchmark Q-Learning on the true environment not provided in the experimental results?\n\n4. In line 201, page 6, section 3.1, what does i.i.d sampling of probability distributions p_i mean?\n\n5. What is the computational overhead of the proposed methodology, and how well will it scale?\n\nLIMITATIONS:\nThe authors addressed one limitation of their work, i.e., that the behavior policy which generates the real-world data is assumed to be given. In terms of societal impact, there would be no potential negative impact of this work according to the considered ethical criterion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper adopts the Bayesian risk MDP (BRMDP) formulation to train a reinforcement learning agent to be robust against model uncertainty. Infinite-horizon discounted value function of BRMDP is defined by a nested formula, and properties for the value function are derived with VaR and CVaR being the risk functional. Furthermore, Q-learning algorithms for BRMDP are proposed using finite-sample estimators for the Bellman operator, and they are shown to converge to the optimal Q-function under some typical assumptions. Numerical experiments for the proposed risk-averse algorithm is shown and compared with its risk-neutral counterpart and other robust RL algorithms.\n\nSTRENGTHS:\n- The paper defines infinite-horizon discounted value for BRMDP, and Theorem 2.3 provides bounds on the difference between the value function of BRMDP to the value function of the true MDP. It shows that the risk-averse version will converge to the true value function as the number of observed transitions grows.\n\n- Q-learning algorithms for BRMDP are proposed using finite-sample estimators for the Bellman operator with either VaR or CVaR as the risk functional. The Q-learning algorithms are shown to converge to the optimal risk-averse Q-function in a data-conditional sense, and this further implies its convergence to the true optimal Q-function given infinite amount of observations.\n\n- Numerical experiments for the proposed risk-averse algorithm is shown and it outperforms two existing robust Q-learning algorithms.\n\nWEAKNESSES:\n- The BRMDP formulation is motivated to provide robust policies, but there is no discussion on what does \"robustness\" mean in the context of this paper. The only place some kind of robustness measure is mentioned is in the numerical experiments where the variations of performance is brought up in the discussion. But there is no proper definition for the variations and there is only very little discussion.\n\n- Theorem 2.3 is claimed to show the trade-off between robustness and conservativeness, but it is hard to talk about robustness without any definition for robustness. In some sense, the upper bound has the risk level parameter $\\alpha$ which may be viewed as a kind of robustness level, but no analysis is provided on how the risk level $\\alpha$ would affect the behavior of the risk-averse policy and how $\\alpha$ would improve the robustness of the performance in some sense.\n\nQUESTIONS:\n- Can we define the robustness using some metric like performance variations? How does the risk-averse policy compared with other policies in terms of some robustness metrics?\n\n- It is shown in Theorem 2.2 that BRMDP has a unique optimal value function, but it's not clear wether the random data-conditional optimal Q-function in Definition 4.1 exists. Do we have existence for the data-conditional optimal Q-function following similar arguments?\n\nLIMITATIONS:\nlimitations are adequately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel multi-stage Bayesian risk-averse Q-learning algorithm to learn the optimal policy with streaming data. A central difference from existing methods is that they attach a risk functional on the future reward at each stage (nested), rather than just once on the total reward. To arrive at unbiased estimators for the Bellman operators, the authors resort to Monte Carlo. The convergence of the Q-learning algorithm is theoretically shown. The authors include empirical investigation of their method on two toy tasks and show that their method is more flexible than some other worst-case criteria.\n\nSTRENGTHS:\nThe authors propose a technically rigorous addition to the the BRMDP literature. Their empirical results clearly show the utility of per-step risk functionals, and their extensive theoretical results include a number of - technically involved - contributions.\nThe centrepiece here is certainly the proof of Theorem 4.3, i.e. the uniform convergence of the Bellman operator estimator. \nThe paper is well-written and structured, and the resulting method has the potential to generate impact on the safe RL community.\n\nWEAKNESSES:\nWhile the paper convinces through its soundness and wealth of theoretical contributions, the empirical evaluation could feature more (and more complex) environments. \n\nA fundamental weakness seem to be limitations in applying the authors' method to high-dimensional data streams. \nFor example, one weakness is that the authors do not extend their results to Q-learning with non-linear function approximation. I believe this could be done fairly easily, and would lay foundations for their method to be used on high-dimensional data streams.\nAnother weakness to address when moving to high-dimensional data streams seems the relative sample-inefficiency of  MC Bellman operator estimators (an empirical study of how efficient their proposed estimator is for high-dimensional data streams would be insightful).\n\nQUESTIONS:\nMy questions concern possible extensions of this work, which I do not deem required for acceptance.\n\n* Do you think that a more sample-efficient way (as MC is unbiased but generally not efficient) of arriving at unbiased Bellman operator estimators could be attained, e.g. using variational inference? This seems to be required for problems featuring high-dimensional data streams.\n\n* The authors note that an extension to active sampling settings would be interesting. This would seem to require \"deep exploration\", necessitating a more fundamental Bayesian RL approach [1]. Do you see an easier way to make progress in this direction?\n\n[1] Bayesian Bellman Operators, Mattie Fellows et al, NeurIPS 2021\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work, although I would have liked to see a discussion of how their method might scale to high-dimensional data streams. I do not see any negative societal impact arising from this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-29"}