SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $X_{k:N}$  is the  $k$ th order statistic out of  $N$  samples. Both estimators are biased with a constant sample size. Increasing the sample size reduces the bias but can be computationally inefficient. As the posterior distribution concentrates more on the true model, even estimators with small sample sizes can have reduced bias. We use a varying sample size that adapts to the updated posterior distribution. Initially, we set  $N_{s,a}$  as a fixed minimal sample size  $N$ , and decrease or increase  $N_{s,a}$  by 1 at the beginning of each stage based on whether a new observation improves or not the estimate for the transition model on  $(s,a)$ . The multi-stage Bayesian Risk-averse Q-Learning algorithms with VaR (BRQL-VaR) and CVaR (BRQL-CVaR) are presented in Algorithm 1.
In this section, we give the theoretical guarantee of our proposed multi-stage Bayesian risk-averse Q-learning algorithm, which ensures the asymptotic convergence of the learned Q-function to the "optimal" Q-function. We define the random observations from the real environment and all algorithm-related random quantities with respect to a probability space  $(\Omega, \mathcal{F}, \mathbb{P})$ .
Recall that the real-world transition observations are assumed to be obtained from some behavior policy that is not controlled by the agents. This may possibly bring a definition problem of "optimal" Q-function. Indeed, if all streaming data are generated by some greedy policy, then the number of transition observations from some state-action pair can remain small, in which case the Bayesian posterior does not converge and the learned policy is risk-averse yet random since it depends on the Bayesian posterior. As in the offline RL, where the policy is learned based on the initial fixed data set, we also define the optimal Q-function as depending on the given real-world data but differs from pure offline RL in that the given data is streaming and random.
Definition 4.1. (Data-conditional optimal Q-function)
Given an observation process  $\omega = \{(s_{t,i},a_{t,i},s_{t,i}^{\prime})|t = 1,2,\ldots ,i = 1,2,\ldots ,n(t)\}$ , let  $O_{s,a,s'}^\infty (\omega) = \sum_{t = 1}^\infty \sum_{i = 1}^{n(t)}\mathbf{1}\{(s_{t,i},a_{t,i},s_{t,i}^{\prime}) = (s,a,s^{\prime})\}$  and  $O_{s,a}^{\infty}(\omega) = (O_{s,a,s^{\prime}}^{\infty}(\omega))_{s^{\prime}\in S}$  denote the number of transition observations under  $\omega$ . For each  $(s,a)\in S\times \mathcal{A}$ , define the limiting posterior as
$$
\phi_ {s, a} ^ {\omega} = \left\{ \begin{array}{l l} \delta_ {p _ {s, a} ^ {c}} (\cdot) & \text {i f} | | O _ {s, a} ^ {\infty} (\omega) | | = \infty , \\ \operatorname {D i r i c h l e t} \left(\phi_ {s, a} ^ {0} + O _ {s, a} ^ {\infty} (\omega)\right) & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $\delta_{p_{s,a}^c}(\cdot)$  is the Dirac measure centered at true transition probability  $p_{s,a}^c = (p_{s,a}^c(s'))_{s' \in S}$ .  $Q^{\omega,*}$  is the optimal solution to BRMDP with "posterior"  $\phi^\omega$ , which satisfies  $Q^{\omega,*} = \mathcal{T}^{\phi^\omega} Q^{\omega,*}$ . The data-optimal  $Q$ -function  $Q^{\omega,*}$  is a random vector since the observation process is random.
With this data-conditional optimal criterion, we can now prove the convergence of the multi-stage Bayesian risk-averse Q-learning algorithm. A list of notations for different Q-functions can be found in Table 1 of the supplementary material.
Algorithm 1 Multi-stage Bayesian risk-averse Q-learning
Input: State space  $\mathcal{S}$ , action space  $\mathcal{A}$ , reward function  $r$ , termination stage  $T$ , Q-learning update step size  $\{m(t)\}_{t=1}^{T}$ , learning rate  $\{\lambda_{\ell}\}_{\ell}$ , prior distribution  $\{\phi_{s,a}^{0}\}_{s\in S, a\in \mathcal{A}}$ , minimal sample size  $N$ , risk measure  $\rho \in \{\mathrm{VaR}_{\alpha}, \mathrm{CVaR}_{\alpha}\}$  with risk level  $\alpha$ . Initialize  $\phi \gets \phi^0$ ,  $Q(s,a) \gets 0$ ,  $N_{s,a} \gets N \forall s,a$ . for  $t = 1$  to  $T$  do Obtain  $n(t)$  observations  $(s_i, a_i, s_i')$ $i = 1, \ldots, n(t)$ . for  $i = 1$  to  $n(t)$  do for all  $(s,a) \in S \times \mathcal{A}$  do  $\phi_{s,a}^{t}(s') \gets \phi_{s,a}^{t-1}(s') + \# \{(s_i, a_i, s_i') = (s,a,s')\}$  if  $\phi_{s,a}^{t} = \phi_{s,a}^{t-1}$  then  $N_{s,a} \gets N_{s,a} + 1$  else  $N_{s,a} \gets \max \{N_{s,a} - 1,N\}$  end if end for end for for  $\ell = 1$  to  $m(t)$  do  $M \gets \sum_{\tau=1}^{t-1} m(t)$ ,  $\lambda_{t,l} \gets \lambda_{M+\ell}$  for all  $(s,a) \in S \times \mathcal{A}$  do Generate  $p_1, \ldots, p_{N_{s,a}} \sim \phi_{s,a}^t$ $X_i \gets f(p_i|s,a,Q)$ ,  $i = 1,2,\ldots,N_{s,a}$ $Q'(s,a) \gets (1-\lambda_{t,\ell})Q(s,a) + \lambda_{t,\ell} \widehat{T}_{N_{s,a}}^{\phi^t}Q(s,a)$  using (3). end for  $Q(s,a) \gets Q'(s,a) \forall (s,a) \in S \times A$  end for end for Output: Q-function  $\{Q(s,a)\}_{(s,a)}$
To prove the convergence of  $Q_{t}$ , we prove (i) the convergence of  $Q^{\phi^t,*}$  which depends on the convergence of the posterior distribution, and (ii) the convergence of the estimator for the Bellman operator. We summarize two important results in Proposition 4.2 and Theorem 4.3. Due to the page limit, all the proofs are deferred to the supplementary material.
Proposition 4.2. (Convergence of  $Q^{\phi^t,*}$ ) Let  $Q^{\omega,*}$  be defined as in Definition 4.1 and  $\phi^t$  be the posterior distribution obtained at stage  $t$ . Then the optimal  $Q$ -function under posterior distribution  $\phi^t$  converges to  $Q^{\omega,*}$  almost surely. That is,
$$
\lim _ {t \rightarrow \infty} | | Q ^ {\phi^ {t}, *} - Q ^ {\omega , *} | | _ {\infty} = 0 \qquad a l m o s t s u r e l y,
$$
where  $||\cdot ||_{\infty}$  is the entry-wise sup norm.
Compared with Theorem 2.6 which characterizes the difference between the BRMDP and the original MDP by computing a concentration bound, Proposition 4.2 ensures the strong convergence of the BRMDP model to the data-conditional optimal Q-function. The next Theorem 4.3 ensures that the bias of the proposed estimator for the Bellman operator with varying sample sizes converges to zero uniformly in  $Q$ .
Theorem 4.3. (Diminishing bias for Bellman estimator) Denote by  $(k)$  the  $k$ th iteration of the  $Q$ -learning update. Let  $t_k$  be the stage such that  $\sum_{t=1}^{t_k-1} m(t) < k \leq \sum_{t=1}^{t_k} m(t)$ . Let  $N_{s,a}^{(k)}$  denote the sample size in iteration  $k$  and  $\omega_{t_k}$  denote all the past observations until stage  $t_k$ . Denote by  $\widehat{\mathcal{T}}^{(k)} Q(s, a) = \widehat{\mathcal{T}}_{N_{s,a}^{(k)}}^{\phi_{t_k}} Q(s, a)$  as defined in (3). Then, we have almost surely,
$$
\lim_{k\to \infty}\sup_{||Q||_{\infty}\leq \frac{\bar{R}}{1 - \gamma}}\left|\mathbb{E}[\widehat{\mathcal{T}}^{(k)}Q(s,a) - \mathcal{T}^{\phi^{t_{k}}}Q(s,a)|\omega_{t_{k}}]\right| = 0.
$$
Theorem 4.3 provides a uniform convergence of the estimated Bellman estimator, which is crucial to prove Theorem 4.4. Notably, with both streaming data and risk measure, proof of Theorem 4.3 is
much more challenging than existing work on distributionally robust Q-learning, where randomness only comes from the learning process, which refers to the random data generated by the simulator. In our setting, we have mixed randomness coming from both the observation process and learning process, which complicates the analysis and differs from that of pure offline Q-learning. Also, the analysis is different from online Bayesian Q-learning, where the sample complexity results are usually constructed assuming an optimal policy is resolved and deployed instantly in each period. In contrast, we aim to prove the convergence of Q-learning in order to obtain the optimal policy.
Together with Proposition 4.2 and Theorem 4.3, we establish the convergence of Algorithm 1 in Theorem 4.4.
Theorem 4.4. Denote by  $Q_{t}$  the  $Q$ -function given by Algorithm 1 at the end of stage  $t$ . Assume  $T = \infty$ , and the learning rate  $\{\lambda_{\ell}\}_{\ell=1}^{\infty}$  satisfies  $\sum_{\ell=1}^{\infty} \lambda_{\ell} = \infty$ ,  $\sum_{\ell=1}^{\infty} \lambda_{\ell}^{2} < \infty$ . Then, we have almost surely,
$$
\lim _ {t \to \infty} | | Q _ {t} - Q ^ {\omega , *} | | _ {\infty} = 0.
$$
Corollary 4.5. Let  $Q^t$  be defined as in Theorem 4.4 and  $O_{s,a}^\infty$  be defined as in Definition 4.1. Assume  $T = \infty$  and  $||O_{s,a}^\infty ||_\infty = \infty, \forall s \in S, a \in \mathcal{A}$  almost surely. Then
$$
\lim _ {t \to \infty} | | Q ^ {t} - Q ^ {c, *} | | _ {\infty} = 0 a l m o s t s u r e l y,
$$
$$
Q ^ {c, *} (s, a) = \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {s, a} ^ {c} (s ^ {\prime}) [ r (s, a, s ^ {\prime}) + \max  _ {b \in \mathcal {A}} Q ^ {c, *} (s ^ {\prime}, b) ]
$$
is the true optimal  $Q$ -function.
Corollary 4.5 is straightforward from Theorem 4.4, which ensures Algorithm 1 will eventually learn the true optimal Q-function if we can obtain an infinite number of observations for each state-action pair.
5.1 Comparison Baselines:
- BRQL-VaR: our proposed multi-stage Bayesian risk-averse Q-learning algorithm with risk measure VaR;
- BRQL-CVaR: our proposed multi-stage Bayesian risk-averse Q-learning algorithm with risk measure CvaR;
- BRQL-mean: the risk-neutral Bayesian Q-learning function. That is, the risk measure is the expectation taken with respect to the posterior distribution;
- DRQL-KL: distributionally robust Q-learning algorithm with KL divergence (see [14]);
- DRQL-Wass: distributionally robust Q-learning algorithm with Wasserstein distance (see [15]).
5.2 Testing examples
Example 1: Coin Toss. Consider we are playing the following game. Each time we will toss K coins and observe the number of coins that show heads, where the chance of each coin showing heads is unknown. After observing the number of heads in the last toss, we can make a guess about whether the next toss will have more heads or fewer heads. If our guess is right, we can get 1 dollar as the reward, otherwise, we need to pay 1 dollar. We also have the choice of not guessing, in which case we do not pay or get paid. We model this game as a discounted infinite-horizon MDP. The state  $s_t \in S = \{0, 1, \dots, K\}$  denotes the number of heads in  $t$ th toss. The actions space is  $\mathcal{A} = \{-1, 0, 1\}$ , where  $a = 1$  corresponds to guess  $s_{t+1} > s_t$ ,  $a = -1$  corresponds to guess  $s_{t+1} < s_t$ , and  $a = 0$  corresponds to not guess. Hence, the reward function is
$$
r (s, a, s ^ {\prime}) := a \mathbf {1} _ {\{s <   s ^ {\prime} \}} - a \mathbf {1} _ {\{s > s ^ {\prime} \}} - | a | \mathbf {1} _ {\{s = s ^ {\prime} \}}.
$$
Assume each coin shows a head with probability 0.4. The number of coins  $K = 10$ .
Example 2: Inventory Management. Suppose a warehouse manager runs a capacity system. At the beginning of period  $t$ , the manager observes the current inventory level  $s_t$  and orders additional goods of the amount  $a_t$ . An ordering cost of  $c = 1$  is incurred for each unit of goods. The demand follows a truncated Poisson distribution with a mean of 3 and support  $\{0, 1, \dots, K\}$ . Suppose the demand arrives during each period and is fulfilled at the end of the period. For each unit of fulfilled demand, we can obtain a profit of  $u = 5$ . If a unit of demand is not fulfilled at the end of the period, the demand is lost and a penalty cost of  $q = 2$  is incurred. If there are any remaining goods at the end of the period, the goods will be taken to the next period with a holding cost  $h = 1$  per unit. The warehouse has a maximal capacity of  $K$  for the goods. We model this problem as a discounted infinite-horizon MDP with state space  $S = \{-K, \dots, 0, \dots, K\}$ , where  $(s_t)^+ = \max(s_t, 0)$  represents the inventory level at the beginning of period  $t$  and  $(s_t)^- = -\min(s_t, 0)$  represents the lost demand at the end of period  $t - 1$ . The action space is  $\mathcal{A}_s := \{0, \dots, K - s^+\}$ . The reward is
$$
r (s, a, s ^ {\prime}) = - \left(c \cdot a + h \cdot (s ^ {\prime}) ^ {+} + q \cdot (s ^ {\prime}) ^ {-}\right) + u \cdot (s ^ {+} + a - (s ^ {\prime}) ^ {+}).
$$
We consider two settings: (I) the demand in each period uniformly distributes among  $\{0,1,\dots ,K\}$  and (II) the demand depends on the current inventory level  $s$ . For the second setting, we will consider the case where observations are insufficient to estimate the transition probability for every state-action pair.
5.3 Experiment Setting
**Coin toss.** We consider stage-wise streaming observations with batch size  $n(t) = 1$  and stage-wise Q-learning with number of steps  $m(t) = 1$ . The minimal sample size to estimate the Bellman operator  $N = 10$ . Initial observed data batch size  $n(0) = 10$ . We set the radius of the KL ball and the Wasserstein ball to be 0.1. The risk level for VaR and CVaR is set to 0.2 in Figure 2 and 0.4 in Figure 3.
Inventory Management. In Figure 4, We set  $K = 10$ ,  $T = 60$ ,  $m(t) = n(t) = 5$ ,  $n(0) = 20$ , and  $N = 10$ . The radius of the KL and the Wasserstein ball is 0.05, and the risk level for VaR and CVaR is 0.2. In Figure 5, we set the capacity  $K = 10$  and the size of the historical data set  $n(0) = 30$ . The policies are deployed in different environments, where the demand follows different truncated Poisson distributions with means ranging from 3 to 5.
5.4 Results
In Figure 2-4, we compare the value functions of different algorithms as the time stage increases. The value function of each algorithm is calculated by deploying the optimal policy in the real environment. Each curve shows the empirical expected performance and the strip around the curve shows the  $95\%$  confidence interval. In both examples, our proposed algorithms outperform the two distributionally robust Q-learning algorithms in both expected performance and variation as the time stage increases, since our proposed algorithms dynamically update the posterior to reduce the model uncertainty while two DRQL algorithms learn with fixed ambiguity set. Compared with the risk-neutral algorithm, BRQL-VaR and BRQL-CVaR achieve lower expected value functions but have smaller variations, which shows the robustness of our two proposed algorithms.
Moreover, in Figure 5 We test for the insufficient data setting, where we only have a set of historical data to estimate the transition model at the initial time stage. The value function is calculated as deploying the learned policy in different environments with demand following different Poisson distributions. Figure 5 indicates the two proposed algorithms achieve higher value functions than the risk-neutral algorithm in the more adversarial setting (with Poisson parameter less than 3), showing their robustness. They also obtain lower value functions than two DRQL algorithms in the adversarial setting and higher value functions in other settings, indicating the risk measure is more flexible compared to the worst-case criterion.
6 Conclusion and Limitation
In this paper, we propose a novel multi-stage Bayesian risk-averse Q-learning algorithm to learn the optimal policy with streaming data, by reformulating the infinite-horizon MDP with unknown transition model as an infinite-horizon BRMDP. In particular, we consider the two cases of risk
Figure 2: Coin Toss: risk level  $\alpha = 0.2$
Figure 3: Coin Toss: risk level  $\alpha = 0.4$
Figure 4: Inventory Management: streaming observations
Figure 5: Inventory Management: fixed BR-MDP
measures, VaR and CVaR, for which we design a Monte Carlo estimator with varying sample sizes to approximate the Bellman operator of BRMDP. We demonstrate the correctness of the BRMDP formulation by providing statistical guarantee and prove the strong asymptotic convergence of the proposed Q-Learning algorithm. The numerical results demonstrate that the proposed algorithms are efficient with streaming data and robust with limited data.
As discussed in the paper, one limitation of the current framework is that the behavior policy that generates real-world observations is assumed to be given. This is suitable for situations when it is expensive for the agent to interact with the real environment or to change the policy frequently as it may cause a large cost or system instability. An interesting future direction is to consider an online learning setting, where at each period the agent also needs to take action in the real world.
ACKNOWLEDGMENT
The authors gratefully acknowledge the support by the Air Force Office of Scientific Research under Grants FA9550-19-1-0283 and FA9550-22-1-0244 and the National Science Foundation under Grant NSF-DMS2053489.
AUTHOR BIOGRAPHIES
Yuhao Wang is a Ph.D. student at the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. He received his B.S. degree from the Department of Mathematics at Nanjing University, China, in 2021. His research interests include simulation and stochastic optimization and reinforcement learning. His email address is yuhaowang@gatech.edu and his web page is https://sites.gatech.edu/yuhaowang/.
Enlu Zhou is a Professor at the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. She received the B.S. degree with highest honors in electrical engineering from Zhejiang University, China, in 2004, and received the Ph.D. degree in electrical engineering from the University of Maryland, College Park, in 2009. Her research interests include
simulation optimization, stochastic optimization, and stochastic control. Her email address is enlu.zhou@isye.gatech.edu, and her web page is https://www.enluzhou.gatech.edu/.
[1] Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203-228, 1999.
[2] Jose H Blanchet, Peter W Glynn, and Yanan Pei. Unbiased multilevel monte carlo: Stochastic optimization, steady-state simulation, quantiles, and other applications. arXiv preprint arXiv:1904.09929, 2019.
[3] Pablo Samuel Castro and Doina Precup. Using linear programming for bayesian exploration in markov decision processes. In IJCAI, volume 24372442, 2007.
[4] Richard Dearden, Nir Friedman, and David Andre. Model-based bayesian exploration. arXiv preprint arXiv:1301.6690, 2013.
[5] Erick Delage and Shie Mannor. Percentile optimization for markov decision processes with parameter uncertainty. Operations research, 58(1):203-213, 2010.
[6] Joseph L Doob. Application of the theory of martingales. Le calcul des probabilités et ses applications, pages 23-27, 1949.
[7] Michael O Duff. Monte-carlo algorithms for the improvement of finite-state stochastic controllers: Application to bayes-adaptive markov decision processes. In International Workshop on Artificial Intelligence and Statistics, pages 93-97. PMLR, 2001.
[8] Michael O'Gordon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. University of Massachusetts Amherst, 2002.
[9] Yaakov Engel, Shie Mannor, and Ron Meir. Bayes meets bellman: The gaussian process approach to temporal difference learning. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 154–161, 2003.
[10] Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[11] Mohammad Ghavamzadeh and Yaakov Engel. Bayesian policy gradient algorithms. Advances in neural information processing systems, 19, 2006.
[12] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 8(5-6):359-483, 2015.
[13] Yifan Lin, Yuxuan Ren, and Enlu Zhou. Bayesian risk markov decision processes. In Advances in Neural Information Processing Systems, 2022.
[14] Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust  $q$ -learning. In International Conference on Machine Learning, pages 13623-13643. PMLR, 2022.
[15] Ariel Neufeld and Julian Sester. Robust  $q$ -learning algorithm for markov decision processes under Wasserstein uncertainty. arXiv preprint arXiv:2210.00898, 2022.
[16] Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
[17] Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics, pages 9582-9602. PMLR, 2022.
[18] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete bayesian reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pages 697-704, 2006.
[19] Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. Advances in Neural Information Processing Systems, 34:1142–1154, 2021.
[20] Stephane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive pomdps. Advances in neural information processing systems, 20, 2007.
[21] Alexander Shapiro. Interchangeability principle and dynamic equations in risk averse stochastic programming. Operations Research Letters, 45(4):377-381, 2017.
[22] Apoorva Sharma, James Harrison, Matthew Tsao, and Marco Pavone. Robust and adaptive planning under model uncertainty. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pages 410-418, 2019.
[23] Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust policy evaluation and learning in offline contextual bandits. In International Conference on Machine Learning, pages 8884-8894. PMLR, 2020.
[24] Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning, 38(3):287-308, 2000.
[25] Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volume 2000, pages 943-950, 2000.
[26] Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Massachusetts Amherst, 1984.
[27] Tao Wang, Daniel Lizotte, Michael Bowling, and Dale Schuurmans. Bayesian sparse sampling for on-line reward optimization. In Proceedings of the 22nd international conference on Machine learning, pages 956-963, 2005.
[28] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
[29] Di Wu, Helin Zhu, and Enlu Zhou. A bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics. SIAM Journal on Optimization, 28(2):1588-1612, 2018.
[30] Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics, 50(6):3223-3248, 2022.
[31] Enlu Zhou and Wei Xie. Simulation optimization when facing input uncertainty. In 2015 Winter Simulation Conference (WSC), pages 3714-3724. IEEE, 2015.
[32] Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331-3339. PMLR, 2021.
A Dirichlet Posterior on State-action Pair
A Dirichlet distribution is parameterized by a count vector  $\phi = (\phi_1, \ldots, \phi_k)$ , where  $\phi_i \geq 1$ , such that the density of probability distribution  $p = (p_1, \ldots, p_k)$  is defined as  $f(p \mid \phi) \propto \prod_{i=1}^{k} p_i^{\phi_i - 1}$ . For each  $(s, a) \in S \times A$ , we impose a Dirichlet prior with parameter  $\phi_{s,a} = (\phi_{s,a}(s'))_{s' \in S}$  on the unknown transition probability  $p_{s,a} = (p_{s,a}(s'))_{s' \in S}$ . After we observe the transition  $(s, a, s')$  for  $o_{s,a,s'}$  times  $\forall s' \in S$ , the posterior distribution of  $p_{s,a}$  is also a Dirichlet distribution with parameter  $\phi_{s,a} + \mathbf{o}_{s,a} = (\phi_{s,a}(s') + o_{s,a,s'})_{s' \in S}$ , where  $\mathbf{o}_{s,a} = (o_{s,a,s'})_{s' \in S}$ .
B Technical Proofs