SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuhao Wang
School of Industrial and Systems Engineering
Atlanta, GA 30332
yuhaowang@gatech.edu
Enlu Zhou
enlu.zhou@isye.gatech.edu
We consider a robust reinforcement learning problem, where a learning agent learns from a simulated training environment. To account for the model misspecification between this training environment and the real environment due to lack of data, we adopt a formulation of Bayesian risk MDP (BRMDP) with infinite horizon, which uses Bayesian posterior to estimate the transition model and impose a risk measure to account for the model uncertainty. Observations from the real environment that are out of the agent's control arrive periodically and are utilized by the agent to update the Bayesian posterior to reduce model uncertainty. We theoretically demonstrate that BRMDP balances the trade-off between robustness and conservativeness, and we further develop a multi-stage Bayesian risk-averse Q-learning algorithm to solve BRMDP with streaming observations from real environment. The proposed algorithm learns a risk-averse yet optimal policy that depends on the availability of real-world observations. We provide a theoretical guarantee of strong convergence for the proposed algorithm.
Markov Decision Process (MDP) is widely applied in the Reinforcement Learning (RL) community to model sequential decision-making, where the underlying transition process is a Markov Process for a given policy. The Q-learning algorithm, which was first proposed in [28], learns the optimal Q-function of the infinite-horizon MDP. The Q-function is a function in terms of the state and action pair  $(s,a)$ , which denotes the expected reward when action  $a$  is taken at the initial state  $s$  and optimal action is taken at subsequent stages. In a Q-learning algorithm, the decision maker updates the value of the Q-function by constantly interacting with the training environment and observing the reward and transition each time. The training environment is often the estimate of the real environment (where the policy is actually implemented) through past observed data. Implementing the policy learned from the training environment in the real environment directly can be risky, because of model misspecification, which is caused by the lack of historical data and possible environment shift. For example, in an inventory management problem where the demand follows some unknown distribution, we have limited observations of past demand realizations or only partially observed data if unfulfilled lost orders are not observed.
One common approach is to consider a distributionally robust formulation, where one assumes the embedding unknown distribution belongs to some ambiguity set. This ambiguity set is usually chosen as a neighborhood of the reference distribution (estimated from data). For example, [23] considered the distributionally robust policy learning for the contextual bandit problems, and [32, 17, 30] studied the distributionally robust policy learning for the discounted RL setting. In particular, [14] and [15] considered Q-learning methods for distributionally robust RL with a discounted reward, where [14] constructed the ambiguity set using the Kullback-Leibler (KL) divergence whereas [15] used the
Wasserstein distance; Both works designed the Q-learning algorithm by reformulating the robust Bellman equation in its dual form, and thus, transforming the optimization over the distributions to the optimization over a scalar.
While distributionally robust formulation accounts for model misspecification and provides a robust policy that has relatively good performance over all possible distributions in the ambiguity set, it is also argued to be conservative as a price for robustness, especially when the worst-case scenario is unlikely to happen in reality. This motivates us to take a more flexible risk quantification criterion instead of only considering the worst-case performance. [31, 29] proposed a Bayesian risk optimization (BRO) framework, which adopts the Bayesian posterior distribution, as opposed to an ambiguity set, to quantify distributional parameter uncertainty. With this quantification, a risk measure, which represents the user's attitude towards risk, is imposed on the objective with respect to the posterior distribution. In this paper, we take this Bayesian risk perspective and formulate the infinite-horizon Bayesian risk MDP (BRMDP) with discounted reward.
BRMDP was first proposed in [13], where they model the unknown parameter of the MDP via a Bayesian posterior distribution. The posterior distribution is absorbed as an augmented state, together with the original physical state, and is updated at each stage through the realization of the reward and transition at that stage. A risk measure is imposed on the future reward at each stage taken with respect to the posterior distribution at that stage. Before [13], using the Bayesian technique in MDP was considered in [8] to deal with parameter uncertainty, where they simply took expectation on the total reward with respect to the posterior distribution and referred to this MDP model as Bayes-adaptive MDP (BAMDP). Based on BAMDP, there is a stream of work on model-based Bayesian RL (e.g., [25, 7, 27, 18, 3, 20, 4, 16]). Other model-free Bayesian RL methods include Gaussian process temporal difference learning [9], Bayesian policy gradient methods [11], and Bayesian actor-critic methods [26]. We referred to [12] for a comprehensive overview of Bayesian RL. More recently, [22] and [19] considered risk-sensitive BAMDP by imposing the risk measure on the total reward. In addition, [5] considered a percentile criterion and formulated a second-order cone programming by assuming a Gaussian random reward. All three works mentioned above took a non-nested formulation i.e., only one risk measure is imposed on the total reward. As pointed out in [13], one benefit of the nested formulation (i.e., a risk measure is imposed at each stage) is that the resulting optimal policy is time consistent, meaning the policy solved at the initial stage remains optimal at any later stage. In contrast, the non-nested formulation does not have this property (see [21]). In this paper, we considered the nested formulation as in [13]. Notably, [13] as well as most works of BAMDP considered the planning problem over a finite horizon. The problem can be solved efficiently only when the number of horizons is small since the number of augmented states, i.e., possible posterior distributions, increases exponentially as the time horizon increases. In this paper, we formulate the infinite-horizon BRMDP whose state space only contains the physical states. As a result, this enables us to find the stationary (time-invariant) optimal policy. We develop a Q-learning algorithm to learn this stationary optimal policy, which is quite different from the dynamic programming approach taken by [13].
On a related note, risk measures are also widely applied in the literature on safe RL (see [10] for a recent survey). In safe RL, risk measures are used to ensure that the learned policy not only performs well but also avoids dangerous states or actions that can cause large one-stage costs. Risk measures are typically applied to deal with the intrinsic uncertainty of the MDP, taking into account the known transition probability or reward distribution. In contrast, our work uses risk measures to account for parametric uncertainty in the transition probability, which can prevent finding the optimal policy. Moreover, in safe RL, the risk measure is usually imposed on some cost function to satisfy some safety constraint, while we impose it on the reward objective. Although both areas use risk measures, they have different goals and frameworks.
To summarize, all the pre-mentioned work on robust RL has focused on offline learning where the agent does not interact with the real environment, but only has access to data from the training environment. In contrast, our work utilizes real-world data to update the Bayesian posterior in the offline learning process to reduce the parametric uncertainty. We also differ from Bayesian online learning, where the Bayesian posterior is updated periodically, and another optimal policy is then re-solved (regardless of computational cost) and deployed. Our approach is off-policy learning, where the learning agent receives streaming real-world observations from some behavior policy, and we focus on designing a Q-learning algorithm for the varying BRMDP that possesses statistical validity rather than simply assuming optimality is obtained after each posterior update.
2 Preliminary and Problem Statement
2.1 Standard Reinforcement Learning
Consider an RL environment  $\mathcal{M}^c = (\mathcal{S},\mathcal{A},\mathcal{P}^c,r,\gamma)$ , where  $\mathcal{S}$  is a finite state space,  $\mathcal{A}$  is a finite action space,  $r:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathbb{R}$  is the reward function bounded by  $\bar{R} = \max_{s,a,s'}|r(s,a,s')|$ ,  $\mathcal{P}^c = \{p_{s,a}^c (\cdot)\}_{(s,a)\in \mathcal{S}\times \mathcal{A}}$  is the transition probabilities, and  $\gamma$  is the discount factor. The standard reinforcement learning aims to find a (deterministic) optimal policy  $\pi :\mathcal{S}\rightarrow \mathcal{A}$  satisfying
$$
V ^ {\pi} (s) = \mathbb {E} _ {s ^ {\prime} \sim p _ {s, \pi (s)} ^ {c}} [ r (s, \pi (s), s ^ {\prime}) + \gamma V ^ {\pi} (s ^ {\prime}) ] = \sup _ {a \in \mathcal {A}} \left\{\mathbb {E} _ {s ^ {\prime} \sim p _ {s, a} ^ {c}} [ r (s, a, s ^ {\prime}) + \gamma V ^ {\pi} (s ^ {\prime}) ] \right\}.
$$
2.2 Bayesian Risk Markov Decision Process (BRMDP)
The true transition probabilities,  $\mathcal{P}^c$ , are usually unknown in real problems, and we need to estimate them using observations from the real world. However, as discussed before, model misspecification due to lack of data can impair the performance of learned policy when deployed in the real environment. Hence, we take a Bayesian approach to estimate the environment and impose a risk measure on the objective with respect to the Bayesian posterior to account for this model (parametric) uncertainty. With finite state and action spaces, we can impose a Dirichlet conjugate prior  $\phi_{s,a}$  on the transition model of each state-action  $(s,a)$  pair and update the Bayesian posterior once we observe a transition from state  $s$  to  $s'$  by taking action  $a$ . We defer the details of updating the Dirichlet posterior on  $(s,a)$  to the supplementary material.
The Bayesian posterior provides a quantification of uncertainty about the transition model, which a risk-sensitive decision maker seeks to address by making robust decisions using the current model estimate. This can be done by imposing a risk measure on the future reward at each stage, which maps a random variable to a real value and reflects different attitudes toward risk. Given a Dirichlet posterior  $\phi = \{\phi_{s,a}\}_{s,a}$  and a risk measure  $\rho_{\xi}(f(\xi))$  that maps a function  $f$  of the random vector  $\xi$  to a real value, the value function of BRMDP under a policy  $\pi$  is defined as
$$
\begin{array}{l} V ^ {\phi , \pi} (s _ {0}) = \mathbb {E} _ {d _ {0} \sim \pi (s _ {0})} \left\{\rho_ {p _ {1} \sim \phi_ {s _ {0}, d _ {0}}} \left(\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s _ {0}, d _ {0}, s _ {1}) + \right. \right. \\ \gamma \mathbb {E} _ {d _ {1} \sim \pi (s _ {1})} \left\{\rho_ {p _ {2} \sim \phi_ {s _ {1}, d _ {1}}} \left(\mathbb {E} _ {s _ {2} \sim p _ {2}} [ r (s _ {1}, d _ {1}, s _ {2}) + \right. \right. \tag {1} \\ \gamma \mathbb {E} _ {d _ {2} \sim \pi (s _ {2})} \left\{\rho_ {p _ {3} \sim \phi_ {s _ {2}, d _ {2}}} \left(\mathbb {E} _ {s _ {3} \sim p _ {3}} [ r (s _ {2}, d _ {2}, s _ {3}) + \dots \right. \right. \\ \end{array}
$$
We assume the risk measure  $\rho$  satisfies the following assumption.
Assumption 2.3. Let  $\xi \in \mathbb{P}(\mathbb{R}^n)$  denote a random vector taking values in  $\mathbb{R}^n$ , and let  $f_{i}:\mathbb{R}^{n}\to \mathbb{R}$ ,  $i = 1,2$  denote two measurable functions. The risk measure satisfies the following conditions:
1.  $\rho_{\xi}(\gamma f_1(\xi)) = \gamma \rho (f_1(\xi))$  for all  $\gamma \geq 0$
2.  $\rho_{\xi}(f_1(\xi))\geq \rho_{\xi}(f_2(\xi))$  if  $f_{1}(\xi)\geq f_{2}(\xi)$  almost surely.
3.  $\rho_{\xi}(f_1(\xi) + C) = \rho_{\xi}(f_1(\xi)) + C$  for all constant  $C$ .
risk measures satisfying Assumption 2.3 are similar to the coherent risk measures (see [1]), except that the sub-additivity is not required. We relax the assumption of coherent risk measure to include the risk measure VaR, which does not admit the sub-additivity.
Notice (1) takes a nested formulation, where the risk measure is imposed on the future reward at each stage as opposed to the non-nested formulation where the risk measure is only imposed once on the total reward, i.e., the value function of the classic MDP. One advantage of the nested formulation is that (1) can be expressed in a recursive form:
$$
V ^ {\phi , \pi} (s) = \mathbb {E} _ {a \sim \pi (s)} \left\{\rho_ {p \sim \phi_ {s, a}} \left(\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V ^ {\phi , \pi} (s ^ {\prime}) ]\right) \right\}.
$$
This enables us to define the Bellman operator to find the time-invariant optimal policy. Let  $\pi^{*}$  denote the optimal policy such that  $V^{\phi, *} \coloneqq V^{\phi, \pi^{*}}$  satisfying
$$
\begin{array}{l} V ^ {\phi , *} (s _ {0}) = \sup  _ {\pi \in \Pi} \left\{\mathbb {E} _ {d _ {0} \sim \pi (s _ {0})} \left\{\rho_ {p _ {1} \sim \phi_ {s _ {0}}, d _ {0}} \left(\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s _ {0}, d _ {0}, s _ {1}) + \right. \right. \right. \\ \gamma \mathbb {E} _ {d _ {1} \sim \pi (s _ {1})} \left\{\rho_ {p _ {2} \sim \phi_ {s _ {1}, d _ {1}}} \left(\mathbb {E} _ {s _ {2} \sim p _ {2}} [ r (s _ {1}, d _ {1}, s _ {2}) + \right. \right. \\ \gamma \mathbb {E} _ {d _ {2} \sim \pi (s _ {2})} \left\{\rho_ {p _ {3} \sim \phi_ {s _ {2}, d _ {2}}} \left(\mathbb {E} _ {s _ {3} \sim p _ {3}} [ r (s _ {2}, d _ {2}, s _ {3}) + \dots \right. \right. \\ \end{array}
$$
where  $\Pi$  contains all randomized (and deterministic) policies. Let  $\mathcal{L}^{\phi}$  be the Bellman operator such that
$$
\mathcal {L} ^ {\phi} V (s) = \max  _ {a \in \mathcal {A}} \rho_ {p \sim \phi_ {s, a}} \left(\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V (s ^ {\prime}) ]\right).
$$
The following theorem ensures that  $\mathcal{L}^{\phi}$  is a contraction mapping and BRMDP admits an optimal value function which is the unique fixed point of  $\mathcal{L}^{\phi}$ . Its proof is in the supplementary material.
Theorem 2.4. BRMDPs possess the following properties:
1.  $\mathcal{L}^{\phi}$  is a contraction mapping with  $||\mathcal{L}^{\phi}V - \mathcal{L}^{\phi}U||_{\infty}\leq \gamma ||V - U||_{\infty}$ , where  $||\cdot ||_{\infty}$  is the sup norm in  $\mathbb{R}^{|\mathcal{S}|}$ .
2. There exists a unique  $V^{\phi, *}$  such that  $\mathcal{L}^{\phi} V^{\phi, *} = V^{\phi, *}$ . Moreover,
$$
\begin{array}{l} V ^ {\phi , *} (s _ {0}) = \sup  _ {\pi \in \Pi} \{\mathbb {E} _ {d _ {0} \sim \pi (s _ {0})} [ \rho_ {p _ {1} \sim \phi_ {s _ {0}, d _ {0}}} (\mathbb {E} _ {s _ {1} \sim p _ {1}} [ r (s _ {0}, d _ {0}, s _ {1}) + \\ \gamma \mathbb {E} _ {d _ {1} \sim \pi (s _ {1})} [ \rho_ {p _ {2} \sim \phi_ {s _ {1}, d _ {1}}} (\mathbb {E} _ {s _ {2} \sim p _ {2}} [ r (s _ {1}, d _ {1}, s _ {2}) + \\ \gamma \mathbb {E} _ {d _ {2} \sim \pi (s _ {2})} [ \rho_ {p _ {3} \sim \phi_ {s _ {2}, d _ {2}}} (\mathbb {E} _ {s _ {3} \sim p _ {3}} [ r (s _ {2}, d _ {2}, s _ {3}) + \dots \\ \end{array}
$$
2.5 BRMDP with VaR and CVaR
Among choices of risk measures that satisfy Assumption 2.3, we adopt two of the most commonly used risk measures, VaR and CVaR. For a random variable  $X$  properly defined on some probability space,  $\mathrm{VaR}_{\alpha}$  is the  $\alpha$ -quantile of  $X$ ,  $\mathrm{VaR}_{\alpha}(X) = \inf \{z|\mathbb{P}(X\leq z)\geq \alpha \}$ , and  $\mathrm{CVaR}_{\alpha}$  normally is defined as the mean of  $\alpha$ -tail distribution of  $X$ . However, since we are imposing the risk measure on the reward (regarded as the negative of loss) rather than the loss, we define  $\mathrm{CVaR}_{\alpha}(X) = \frac{1}{\alpha}\int_{-\infty}^{\mathrm{VaR}_{\alpha}(X)}x\mathbb{P}(\mathrm{d}x)$ , which is the conditional expectation for  $X\leq \mathrm{VaR}_{\alpha}$ .
Before discussing solving the BRMDP, a natural question is how this risk-averse formulation is related to the original risk-neutral objective. Let  $V^{c,\pi}$  denote the value function of policy  $\pi$  under the true MDP  $\mathcal{M}^c$ . The difference between  $V^{\phi ,\pi}$  and  $V^{c,\pi}$  is bounded by the following theorem.
Theorem 2.6. Suppose the Bayesian prior  $\phi_{s,a}^{0}(s^{\prime}) = 1, \forall s, a, s^{\prime}$  and  $\rho$  is either  $VaR_{\alpha}$  or  $CVaR_{\alpha}$ . Let  $\pi$  be a deterministic policy. Let  $\bar{O} = \min_{s \in S} O_{s,\pi(s)}$  and  $O_{s,a}$  be the number of observed transitions from  $s$  with action  $\pi(s)$  in the dataset. Assuming  $O_{s,a} > 0$  for all  $(s,a)$ , then with probability at least  $1 - \bar{O}^{-\frac{1}{3}} \sqrt{\frac{|S|}{\alpha}}$ ,
$$
\| V ^ {\phi , \pi} - V ^ {c, \pi} \| _ {\infty} \leq \bar {O} ^ {- \frac {1}{3}} \sqrt {\frac {| S |}{\alpha}} \frac {5 | S | \bar {R}}{(1 - \gamma) ^ {2}},
$$
In particular, let  $\underline{O} = \min_{s\in S,a\in \mathcal{A}}O_{s,a}$ , then with probability at least  $1 - \underline{O}^{-\frac{1}{3}}\sqrt{\frac{|S|}{\alpha}}$
$$
\| V ^ {\phi , *} - V ^ {c, *} \| _ {\infty} \leq \underline {{O}} ^ {- \frac {1}{3}} \sqrt {\frac {| S |}{\alpha}} \frac {5 | S | \bar {R}}{(1 - \gamma) ^ {2}}
$$
where  $V^{c,*}$  is the optimal value function for  $\mathcal{M}^c$
Theorem 2.6 implies our reformulated BRMDP coincides with the risk-neutral MDP as more observations are available. In particular, the discrepancy is characterized in terms of the minimal number of observations for any state-action pair with the order at least  $\underline{O}^{-\frac{1}{3}}$ . This indicates the BRMDP automatically balances the trade-off between robustness and conservativeness with varying numbers of observations. We defer the proof to the supplementary material due to the space limit.
2.7 Q-Learning for BRMDP
Section 2.2 ensures BRMDP is well-formulated, that is, it admits an optimal value function which can be found as the fixed point of its Bellman operator  $\mathcal{L}^{\phi}$ . This allows us to derive a Q-Learning algorithm to learn the optimal policy and value function of BRMDP. Recall that an optimal Q-function is defined as
$$
Q ^ {\phi , *} (s, a) = \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma V ^ {\phi , *} (s ^ {\prime}) ]) = \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma \max _ {b \in \mathcal {A}} Q ^ {\phi , *} (s ^ {\prime}, b) ]).
$$
Let  $\mathcal{T}^{\phi}$  denote the Bellman operator for the Q-function. That is,
$$
\mathcal {T} ^ {\phi} Q (s, a) = \rho_ {p \sim \phi_ {s, a}} (\mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma \max _ {b \in \mathcal {A}} Q (s ^ {\prime}, b) ]).
$$
The optimal Q-function  $Q^{\phi, *}(s, a)$  then satisfies
$$
Q ^ {\phi , *} (s, a) = (1 - \lambda) Q ^ {\phi , *} (s, a) + \lambda \mathcal {T} ^ {\phi} Q ^ {\phi , *} (s, a),
$$
where  $\lambda \in (0,1)$ . In a Q-learning algorithm, given a sequence of learning rates  $\{\lambda_t\}_t$  and an estimator of the Bellman operator  $\widehat{T}^{\phi}$ , we have the Q-learning update rule
$$
Q _ {t + 1} (s, a) = \left(1 - \lambda_ {t}\right) Q _ {t} (s, a) + \lambda_ {t} \widehat {\mathcal {T}} ^ {\phi} Q _ {t} (s, a).
$$
2.8 Bayesian Risk-Averse Q-Learning with Streaming Data
Notice that in Section 2.2 and 2.7, the posterior distribution  $\phi$  is fixed across stages. As in many previous works, the embedding model is estimated with a fixed set of past observations before solving the problem. However, this one-time estimation of the transition model does not take into consideration utilizing the new data that arrive later, which helps reduce the model uncertainty as well as the conservativeness caused by risk measure, as indicated by Theorem 2.6. This motivates us to consider a data-driven framework where we dynamically update the estimate of the model. For this purpose, we consider a multi-stage Bayesian Q-learning algorithm.
Suppose at the beginning of stage  $t$ , a batch of observations in the form of three-tuple  $(s_i, a_i, s_i')$  with batch size  $n(t)$  is available. The observation can be regarded as generated by some policy that is actually deployed in the real world, which does not depend on the learning process. The decision maker incorporates these new data to update the posterior  $\phi^t$ , with which we obtain a BRMDP model  $\mathcal{M}_t$ . We then carry out  $m(t)$  steps of the Q-learning update, where the initial Q-function in stage  $t$  is inherited from the previous stage  $t - 1$ . This framework is shown in Figure 1.
Figure 1: Multi-stage Bayesian risk-averse Q-learning
3 Estimator for Bellman Operator
In Figure 1, a key step is to design a proper estimator  $\widehat{\mathcal{T}}^{\phi}$  for the Bellman operator to ensure the convergence of the Q-function. In most of the existing literature on Q-learning, convergence relies on an unbiased estimator of the Bellman operator. While this is usually easy to obtain with the expectation operator which is linear, in BRMDP unbiased estimator for the Bellman operator is difficult if not impossible to obtain, because of (i) the non-linearity of risk measure (VaR and CVaR) and (ii) varying posterior distributions. Unbiased estimators for nonlinear functionals have been studied in [2]; however, their method cannot be directly applied here since the variance of the estimator is uncontrollable in the existence of varying posteriors. Instead, we use the Monto Carlo simulation to obtain an estimator with almost surely diminishing bias. We show in Theorem 4.3 and 4.4 that the Monto Carlo estimator is sufficient to guarantee the convergence of the Q-function.
3.1 Monto Carlo Estimator for VaR and CVaR with Varying Sample Size
Denote by
$$
f (p \mid s, a, Q) = \mathbb {E} _ {s ^ {\prime} \sim p} [ r (s, a, s ^ {\prime}) + \gamma \max  _ {b \in \mathcal {A}} Q (s ^ {\prime}, b) ] = \sum_ {s ^ {\prime} \in \mathcal {S}} p (s ^ {\prime}) [ r (s, a, s ^ {\prime}) + \gamma \max  _ {b \in \mathcal {A}} Q (s ^ {\prime}, b) ]. \tag {2}
$$
Given a posterior  $\phi$  and a Q-function  $Q$ , we want to estimate
$$
\mathcal {T} ^ {\phi} Q (s, a) = \rho_ {p \sim \phi_ {s, a}} (f (p | s, a, Q)),
$$
where  $\rho$  is either  $\mathrm{VaR}_{\alpha}$  or  $\mathrm{CVaR}_{\alpha}$  and  $\alpha \in (0,1)$  denotes the risk level. A Monte Carlo estimator for  $\mathrm{VaR}_{\alpha}(f(p|s,a,Q))$  and  $\mathrm{CVaR}_{\alpha}(f(p|s,a,Q))$  with sample size  $N$  can be obtained by first drawing independent and identically distributed (i.i.d.) samples  $p_1,p_2,\ldots ,p_N\sim \phi_{s,a}$ . Denote by  $X_{i}:= f(p_{i}|s,a,Q), i = 1,2,\dots ,N$ . Then
$$
\widehat {\mathcal {T}} _ {N} ^ {\phi} Q (s, a) = \left\{ \begin{array}{l l} \widehat {\operatorname {V a R}} _ {\alpha , N} ^ {\phi_ {s, a}} (f (p | s, a, Q)) := X _ {\lceil N \alpha \rceil : N} & \text {i f} \rho \text {i s V a R} _ {\alpha}, \\ \widehat {\operatorname {C V a R}} _ {\alpha , N} ^ {\phi_ {s, a}} (f (p | s, a, Q)) := \frac {1}{\lceil N \alpha \rceil} \sum_ {k = 1} ^ {\lceil N \alpha \rceil} X _ {k: N} & \text {i f} \rho \text {i s C V a R} _ {\alpha}, \end{array} \right. \tag {3}
$$