SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.4 states that  $\gamma$ -quantile of the distribution  $P_{1}$  is the  $(\gamma + \delta_{\gamma, \epsilon})$ -quantile of the distribution  $P_{2}$  where the quantile shift  $\delta_{\gamma, \epsilon}$  has the following properties. a) The upper bound of  $\delta_{\gamma, \epsilon}$  is close to 0 if  $D_{\alpha}(P_{1}, P_{2})$  with  $\alpha > 1$  is bounded; b) The lower bound of  $\delta_{\gamma, \epsilon}$  is close to 0 if  $D_{\alpha}(P_{1}, P_{2})$  with  $\alpha < 0$  is bounded; c) A slightly large bound on  $D_{\alpha}(P_{1}, P_{2})$  with  $0 < \alpha < 1$  cannot control the shift  $\delta_{\gamma, \epsilon}$  in general, which gives the intuition that  $\alpha \in (0, 1)$  is not implemented in Assumption 3.3.
This theorem is distribution-free, in the sense that the bound of  $\delta_{\gamma,\epsilon}$  does not depend on any specific distributions (noting that distribution changes as  $t$  evolves in bandit problems). In particular, a) + b) in Theorem 3.4 shows that  $C_{\epsilon,\alpha_1}(1-\gamma)^{\frac{\alpha_1}{\alpha_1-1}} \geq \delta_{\gamma,\epsilon} - (1-\gamma) \geq -C_{\epsilon,\alpha_2}(1-\gamma)^{\frac{\alpha_2}{\alpha_2-1}}$ , with  $\alpha_1 > 1$  and  $\alpha_2 < 0$  where  $C_{\epsilon,\alpha} = (\epsilon\alpha(\alpha-1)+1)^{\frac{1}{1-\alpha}}$  is independent of  $\gamma$  or distributions, and thus independent of the time step  $t$  in our EBUCB algorithm. This observation is important in the robustness of using quantiles in the EBUCB. The proof of Theorem 3.4 relies on the following lemma, which provides a quantile-based representation of  $\alpha$ -divergence.
Lemma 3.5. Under the same conditions in Theorem 3.4, we have that for any  $\alpha$ -divergence,
$$
D _ {\alpha} (P _ {1}, P _ {2}) = \frac {\int_ {0} ^ {1} \left(\frac {d}{d u} R _ {2} ^ {- 1} (R _ {1} (u))\right) ^ {1 - \alpha} d u - 1}{\alpha (\alpha - 1)}.
$$
3.3 Finite-Time Regret Bound for EBUCB
In this section, we derive a finite-time upper bound of problem-dependent frequentist regret for our EBUCB algorithm in Bernoulli multi-armed bandit problems. Without loss of generality, we assume action 1 is optimal in the subsequent theorems.
To begin with, we first express the frequentist regret as  $R(T,\mathcal{A})\coloneqq \mathbb{E}\left[\sum_{t = 1}^{T}(\mu_1 - \mu_{A_t})\right] = \sum_{j = 1}^{K}(\mu_1 - \mu_j)\mathbb{E}\left[N_j(t)\right]$  by (1). Therefore, it is sufficient to study each term  $\mathbb{E}\left[N_j(t)\right]$  in order to bound the regret  $R(T,\mathcal{A})$ . For  $(p,q)\in [0,1]^2$ , we denote the Bernoulli KL divergence between two points by  $d(p,q) = p\log (\frac{p}{q}) + (1 - p)\log (\frac{1 - p}{1 - q})$ , with  $0\log 0 = 0\log (0 / 0) = 0$  and  $x\log (x / 0) = +\infty$  for  $x > 0$  by convention. We also denote that  $d^{+}(p,q) = d(p,q)\mathbb{1}\{p < q\}$  for convenience.
Note that  $\pi_{t,j}(x)$  is the density of  $\mathrm{Beta}(1 + S_j(t), 1 + N_j(t) - S_j(t))$  so its (closed) support is  $[0,1]$ . We put a basic assumption on the approximate distribution  $Q_{t,j}$ .
Assumption 3.6.  $Q_{t,j}$  has the density  $q_{t,j}(x)$  whose support is  $[0,1]$  for any  $t\in [T],j\in [K]$ .
The following is our main theorem, which establishes a finite-time regret bound for our EBUCB algorithm.
Theorem 3.7. Suppose Assumptions 3.3 and 3.6 hold. Let  $M_{\epsilon,1} = (\epsilon \alpha_1(\alpha_1 - 1) + 1)^{\frac{1}{1 - \alpha_1}} < 1$ ,  $\tilde{\alpha}_1 = \frac{\alpha_1}{\alpha_1 - 1} > 0$ ,  $M_{\epsilon,2} = (\epsilon \alpha_2(\alpha_2 - 1) + 1)^{\frac{1}{1 - \alpha_2}} > 1$ , and  $\tilde{\alpha}_2 = \frac{\alpha_2}{\alpha_2 - 1} > 0$ . For any  $\xi > 0$ , choosing the parameter  $c$  such that  $c\tilde{\alpha}_2 \geq 5$  in the EBUCB algorithm and setting  $\gamma_t = 1 - \frac{1}{t^\zeta(\log T)^c} (\zeta > 0)$ , the number of draws of any sub-optimal action  $j \geq 2$  is upper-bounded by
$$
\mathbb {E} [ N _ {j} (T) ] \leq \frac {\left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2}\right) M _ {\epsilon , 2} e T ^ {1 - \zeta \tilde {\alpha} _ {2}}}{1 - \zeta \tilde {\alpha} _ {2}} + o \left(T ^ {1 - \zeta \tilde {\alpha} _ {2}}\right)
$$
if  $0 < \zeta \tilde{\alpha}_2 < 1$ , and
$$
\mathbb {E} \left[ N _ {j} (T) \right] \leq \frac {(1 + \xi) \zeta \tilde {\alpha} _ {1}}{d (\mu_ {j} , \mu_ {1})} \log (T) + o (\log T)
$$
if  $\zeta \tilde{\alpha}_2\geq 1$
Theorem 3.7 provides an exact finite-time regret bound and the  $o(\cdot)$  term in Theorem 3.7 has an exact finite-time closed-form expression that holds for any time horizon  $T$ ; See Step 4 in the proof of Theorem 3.7 in Appendix C. We only show the most dominant term of the regret bound and shrink the rest to the  $o(\cdot)$  term to improve the readability of the main paper. Note that this bound has explicit dependence on  $\epsilon$ , which is  $M_{\epsilon,1}^{-1}$  and  $M_{\epsilon,2}$  in Step 4 in the proof of Theorem 3.7. Obviously, the error terms  $M_{\epsilon,1}^{-1}$  and  $M_{\epsilon,2}$  in the bound increase as  $\epsilon$  increases. However, this dependence on  $\epsilon$  does not impact the dominating term too much. The exact posterior will be more "concentrated" on the true mean with small variability as the time  $t$  increases, and the impact from the error will vanish; See Remark 3.9.
It is easy to see that to minimize the regret upper bound, we may choose  $\zeta = \frac{1}{\tilde{\alpha}_2}$  in Theorem 3.7.
Corollary 3.8. Under the same conditions in Theorem 3.7, for any  $\xi >0$ , choosing the parameter  $c$  such that  $c\tilde{\alpha}_2\geq 5$  in the EBUCB algorithm and setting  $\gamma_{t} = 1 - \frac{1}{t^{1 / \tilde{\alpha}_{2}}(\log T)^{c}}$ , the number of draws of any sub-optimal action  $j\geq 2$  is upper-bounded by  $\mathbb{E}[N_j(T)]\leq \frac{(1 + \xi)\frac{\tilde{\alpha}_1}{\tilde{\alpha}_2}}{d(\mu_j,\mu_1)}\log (T) + o(\log T)$ .
This result states that with the  $\epsilon$  error threshold, the regret of the EBUCB algorithm is bounded above by  $O(\log T)$  regardless of how large  $\epsilon$  is, which reaches the same order  $(\log T)$  of the problem-dependent frequentist regret lower bound [25]. In comparison with the exact lower bound, there is a slight difference in the multiplier before the order  $(\log T)$ : Our upper bound in Corollary 3.8 has the additional multiplier  $\frac{\tilde{\alpha}_1}{\tilde{\alpha}_2} > 1$ , which arises from the approximate inference when estimating the posterior distributions (Assumption 3.3). If in addition Assumption 3.3 holds for any  $\alpha_1 > 1$  and any  $\alpha_2 < 0$ , then we can let  $\frac{\tilde{\alpha}_1}{\tilde{\alpha}_2} \to 1$  by taking  $\alpha_1 \to +\infty$  and  $\alpha_2 \to -\infty$  to match the exact lower bound.
In the absence of approximate inference, [21] showed that  $\mathbb{E}[N_j(T)] \leq \frac{(1 + \xi)}{d(\mu_j, \mu_1)} \log (T) + o(\log T)$  matching the exact lower bound. Prior to our work, it was unclear in the literature whether the optimal regret order  $O(\log T)$  could be achieved in the presence of bounded approximate inference error. Our result provides a positive answer to this question, despite the fact that the inference error may cause the multiplier  $\frac{\tilde{\alpha}_1}{\tilde{\alpha}_2}$  before the order  $(\log T)$ . To the best of our knowledge, this is the first algorithm providing the regret upper bound that is better than  $o(T)$  in [34] with bounded approximate inference error.
As discussed in [21], the horizon-dependent term  $(\log T)^c$  in Corollary 3.8 is only an artifact of the theoretical analysis to obtain a finite-time regret upper bound. In practice, the model with choice  $c = 0$  (i.e., without the horizon-dependent term) already achieves superior performance. This is confirmed by our experiments in Section 4. A similar observation in BUCB was indicated in [21].
Remark 3.9. It might appear a little surprising that the result in Corollary 3.8 indicates a regret upper bound with the dominating term  $O(\log T)$  that does not depend on  $\epsilon$ , as one may expect that a large  $\epsilon$  allows the "full swap" between the posterior of the optimal action and the posterior of a suboptimal action, making any Bayesian-based approaches unable to distinguish them. However, benefiting from historical observations, the exact posterior will be more "concentrated" on the true mean with small variability, which will keep enlarging the  $\alpha$ -divergence between two actions. This indicates that, for a fixed  $\epsilon$ , the  $\alpha$ -divergence between the exact posteriors of two actions can be sufficiently large as  $t$  increases and ultimately exceeds  $\epsilon$ . Hence, the "full swap" will not happen when  $t$  is large.
Remark 3.10. The  $\frac{1}{t^{1 / \tilde{\alpha}_2}}$  in EBUCB, instead of the original  $\frac{1}{t}$  in BUCB, is a delicate choice to address the tradeoff between making the regret optimal without approximate inference and the presence of inference error. On a technical level, a power  $\zeta$  close to 1 in  $t^{\zeta}$  improves the regret bound without the presence of approximate inference but simultaneously leads to high-level quantile shift caused by approximate inference. Choosing  $\zeta = \frac{1}{\tilde{\alpha}_2}$  is a subtle balance of these two.
The technical derivation of Theorem 3.7 depends on analyzing the quantiles of the approximate distributions used in the EBUCB algorithm. In particular, one of the major techniques in our analysis is Lemma C.1 in Appendix C. It provides explicit upper and lower bounds on the tails of approximate distributions to control the quantiles designed by the EBUCB algorithm. It is obtained by combining the quantile shift between the approximate and exact posterior distributions that developed in Theorem 3.4 (Section 3.2) with the tight bounds on the quantiles of the exact posterior distributions (the proof of Lemma 1 in [21]). This result is then used to bound the expectation of a decomposition of  $N_{j}(T)$  in Lemma C.2 that links  $N_{j}(T)$  to the over-estimation of the optimal arm.
3.4 Negative Results
We show that one bounded  $\alpha$ -divergence alone cannot guarantee a sub-linear regret. We provide two worst-case examples, one where Thompson sampling has a linear regret, and the other where BUCB/EBUCB has a linear regret, even when the inference error measured by one  $\alpha$ -divergence is small. A similar study on Thompson sampling was conducted in [34] with a special focus on the inference error on the joint distribution of all actions. In our study, nevertheless, we focus on a setting where the inference error on the distribution of each action is assumed; See Remark A.3. Therefore, the examples in [34] cannot be directly applied in our setting. Moreover, our second example shows that BUCB/EBUCB could have a linear regret if only one  $\alpha$ -divergence is considered, which is new.
Assumption 3.11. Suppose that there exists a positive value  $\epsilon \in (0, +\infty)$  such that
$$
D _ {\alpha} \left(Q _ {t, j}, \Pi_ {t, j}\right) \leq \epsilon , \forall t \in [ T ], j \in [ K ]. \tag {4}
$$
We establish the following theorem for Thompson sampling:
Theorem 3.12. Consider a Bernoulli multi-armed bandit problem where the number of actions is  $K = 2$  and  $\mu_1 > \mu_2$ . The prior  $\Pi_{0,j}$  is chosen to be the i.i.d. Beta(1,1), or the uniform distribution for every action  $j = 1,2$ . For any given  $\alpha < 1$  and any error threshold  $\epsilon > 0$ , there exists a sequence of distributions  $Q_{t - 1}$  such that for all  $t \geq 1$ :
1) The probability of sampling from  $Q_{t-1}$  choosing action 2 is greater than a positive constant independent of  $t$ .
2)  $Q_{t - 1}$  satisfies Assumptions 3.6 and 3.11.
Therefore Thompson sampling from the approximate distribution  $Q_{t-1}$  will cause a finite-time linear frequentist regret:  $R(T, \mathcal{A}) = \Omega(T)$ .
This theorem shows that making one  $\alpha$ -divergence a small constant alone, even for each action  $j$ , is not sufficient to guarantee a sub-linear regret of Thompson sampling. Note that Theorem 3.12 is an enhancement of the results in [34] in the sense that the  $Q_{t}$  constructed by our theorem satisfies more restrictive assumptions. We can derive a similar observation for the BUCB/EBUCB algorithm as follows:
Theorem 3.13. Consider a Bernoulli multi-armed bandit problem where the number of actions is  $K = 2$  and  $\mu_1 > \mu_2$ . The prior  $\Pi_{0,j}$  is chosen to be the i.i.d. Beta(1,1), or the uniform distribution for every action  $j = 1,2$ . Consider the general EBUCB algorithm described in Algorithm 1. For any given  $\alpha < 1$  and any error threshold  $\epsilon > 0$ , there exists a constant  $T_0$  (only depending on  $\epsilon$ ,  $\alpha$ , and the sequence  $\{\gamma_t\}$ ) and a sequence of distributions  $Q_{t - 1}$  such that for all  $t \geq 1$ :
1) The EBUCB algorithm always chooses action 2 when  $t \geq T_0$ .
2)  $Q_{t - 1}$  satisfies Assumptions 3.6 and 3.11.
Therefore the EBUCB algorithm from the approximate distribution  $Q_{t-1}$  will cause a finite-time linear frequentist regret:  $R(T, \mathcal{A}) = \Omega(T)$ .
This theorem shows that making one  $\alpha$ -divergence a small constant alone, even for each action  $j$ , is insufficient to guarantee a sub-linear regret of BUCB/EBUCB. We emphasize that the examples in Theorems 3.12 and 3.13 are in the worst-case sense, indicating that there exist worst-case examples where Thompson sampling/EBUCB exhibits a linear regret if only one  $\alpha$ -divergence is bounded. However, this does not imply that EBUCB and Thompson sampling would fail on average in the presence of approximate inference. In fact, Theorem 3.7 shows that a sub-linear regret can be achieved if the inference error measured by two different  $\alpha$ -divergences is bounded.
In this section, we conduct numerical experiments to show the correctness of our theory. In Section 4.1, we compare the performance of EBUCB with the following baselines: BUCB (using its originally proposed quantile and  $Q_{t}$  as  $\Pi_t$  since the exact posterior distribution  $\Pi_t$  is unavailable) and Thompson sampling (using  $Q_{t}$  as  $\Pi_t$ ). In Section 4.2, we construct worst-case examples showing that both EBUCB and Thompson sampling can degenerate to linear regret if only one  $\alpha$ -divergence is bounded. We consider the Bernoulli multi-armed bandit problem which has two actions with mean rewards
2The source code for experiments is available at https://github.com/HZ0000/EBUCB.
BUCB  $(c = 5)$
EBUCB  $(c = 5)$
Figure 1: Comparison of EBUCB and baselines with generally misspecified posteriors under different problem settings. Results are averaged over 10 runs with shaded standard errors.
Figure 2: Results of Thompson sampling, BUCB, and EBUCB with worst-case misspecified posteriors under different problem settings. Results are averaged over 10 runs with shaded standard errors.
[0.7, 0.3], and use  $\mathrm{Beta}(1,1)$  as the prior distribution of mean reward for each action. At each time step  $t$ , the exact posterior distribution for each action is  $\mathrm{Beta}(1 + S_j(t),1 + N_j(t) - S_j(t))$ , where  $S_{j}(t)$  is the empirical cumulative reward from action  $j$  up to time  $t$  and  $N_{j}(t)$  is the number of draws from action  $j$  up to time  $t$ .
4.1 Generally Misspecified Posteriors
Suppose the posterior distributions are misspecified to the following distributions:
$(P1):(1 - w)*\mathbf{Beta}(1 + S_j(t),1 + N_j(t) - S_j(t)) + w*\mathbf{Beta}(\frac{1 + S_j(t)}{2},\frac{1 + N_j(t) - S_j(t)}{2})$
$(P2):(1 - w)*\mathrm{Beta}(1 + S_j(t),1 + N_j(t) - S_j(t)) + w*\mathrm{Beta}(2(1 + S_j(t)),2(1 + N_j(t) - S_j(t)))$
where  $w = 0.9, 0.8, 0.7$ . Figure 1 presents the results of EBUCB and the baselines. Overall, EBUCB achieves consistently superior performance than the baselines, and it outperforms BUCB with considerable improvements. These results confirm the effectiveness of EBUCB across multiple settings. Moreover, EBUCB performs well without the horizon-dependent term (i.e.,  $c = 0$  in Corollary 3.8). This brings EBUCB practical advantages in real-world applications, as it does not require advanced knowledge of the horizon (i.e., anytime). A similar observation of BUCB was also noticed in [21].
4.2 Worst-Case Misspecified Posteriors
We consider the worst-case examples, Equations (13) and (14), presented in the proof of Theorems 3.12 and 3.13, where the posterior distributions are misspecified using one  $\alpha$ -divergence. The results of Thompson sampling, BUCB, and EBUCB are displayed in Figure 2. From these worst-case examples, we observe that: 1) Thompson sampling exhibits a linear regret after  $t \geq 1$ . As shown in Theorem 3.12, the linear coefficient (i.e., the slope) of the regret depends on the level  $r$  that corresponds to the inference error. Specifically, the slope of the regret is increased along with the increased value of  $r$ , as illustrated in both Figure 2 and the proof of Theorem 3.12. 2) BUCB/EBUCB exhibits a linear regret with constant slope  $\mu_1 - \mu_2$  after  $t \geq T_0$ , where  $T_0$  is the time threshold introduced in Theorem 3.13 after which BUCB/EBUCB always chooses the sub-optimal action. The artificial choice of  $r$  is to make  $T_0 = 100, 200, 333$  where  $\gamma_{T_0} = \frac{1}{r}$ ; See the proof of Theorem 3.13.
In summary, our experiments evidently demonstrate the superior performance of our proposed EBUCB on multi-armed bandit problems with generally misspecified posteriors. Our results also align closely with our theory that making one  $\alpha$ -divergence a small constant alone is insufficient to guarantee a sub-linear regret of Thompson sampling/BUCB/EBUCB. Hence, making two different  $\alpha$ -divergences bounded is necessary for the sub-linear regret upper bound.
In this paper, we propose a general Bayesian bandit algorithm, Enhanced Bayesian Upper Confidence Bound (EBUCB), that achieves superior performance for Bernoulli bandit problems with approximate inference. We prove that, if the inference error measured by two different  $\alpha$ -divergences is less than a constant, EBUCB can achieve the optimal regret order  $O(\log T)$ . Additionally, we construct worse-case examples to show the necessity of bounding two different  $\alpha$ -divergences, which is further validated by our experiments. We consider the study of other problem settings as meaningful future research that could be built upon our current framework, e.g., extending to the general exponential family bandit problems by leveraging the techniques in [20]. We will also extend our current framework to contextual bandit problems and investigate the performance of contextual Bayesian bandit algorithms with approximate inference [36].
[1] S. Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on Learning Theory, pages 39–1, 2012.
[2] S. Agrawal and N. Goyal. Further optimal regret bounds for thompson sampling. In Artificial Intelligence and Statistics, pages 99-107, 2013.
[3] S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pages 127-135, 2013.
[4] C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan. An introduction to mcmc for machine learning. Machine Learning, 50(1):5-43, 2003.
[5] P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
[6] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-256, 2002.
[47] D. Wang, H. Liu, and Q. Liu. Variational inference with tail-adaptive f-divergence. Advances in Neural Information Processing Systems, 31, 2018.
[48] P. Xu, H. Zheng, E. V. Mazumdar, K. Azizzadenesheli, and A. Anandkumar. Langevin monte carlo for contextual bandits. In International Conference on Machine Learning, pages 24830-24850. PMLR, 2022.
[49] Y. Yang, D. Pati, and A. Bhattacharya.  $\alpha$ -variational inference with statistical guarantees. The Annals of Statistics, 48(2):886 - 905, 2020.
[50] W. Zhang, D. Zhou, L. Li, and Q. Gu. Neural thompson sampling. In International Conference on Learning Representations, 2021.
[51] D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pages 11492-11502. PMLR, 2020.
[52] H. Zhu and R. Rohwer. Information geometric measurements of generalisation. Technical Report. Aston University, Birmingham, UK, 1995.
We provide further results and discussions in this appendix. Section A presents detailed discussions on Assumption 3.3. Section B presents proofs for the results in Section 3.2 in the main paper. Section C presents proofs for the results in Section 3.3. Section D presents proofs for the results in Section 3.4. Section E presents additional experimental results.
Appendix A Remarks on Assumption 3.3
Remark A.1 (Constant Threshold). A constant threshold assumption on inference error appears in [34]. In this study, we adopt a similar assumption as [34], since no standard assumptions are in place due to the infancy of this field. In practice, keeping inference error below a constant threshold is not only feasible but also can be improved. As a concrete example, [30] showed that the inference error of an efficient Langevin MCMC algorithm decreases at the order of  $O(1 / \sqrt{N_j(t)})$ . [48] also establish similar consistency results. On a high level, as more data are collected, the approximate and exact posteriors both become more concentrated, but they could concentrate to the same point of true mean rewards, as [30] showed. In addition to MCMC algorithms [4] which typically produce a consistent posterior, our assumption also fits other Bayesian inference algorithms, such as variational inference [8], that output approximate posteriors with a systematic computational bias.
In fact, the negative results in the presence of constant inference error shown by [34] motivate another research direction: A natural idea for obtaining positive results is to construct a "highly-effective" Bayesian inference approach with vanishing inference error as the number of samples increases, as studied in [30, 48]. However, our results demonstrate that the  $O(\log T)$  regret upper bound is achievable even with non-vanishing inference error, without requiring specific "highly-effective" Bayesian inference approaches.
Remark A.2 ( $\alpha$ -divergence). Studies beyond KL divergence [38, 9], using  $\alpha$ -divergence [26, 19, 14, 49] or more generally  $f$ -divergence [46, 47], have appeared in recent research. These studies demonstrated the potential use of general  $\alpha$ -divergence and other types of divergence in practical model design. Moreover, from a practical perspective, the two  $\alpha$ -divergences between the estimated and the exact posterior distributions could be small, even if only one  $\alpha$ -divergence is used to construct the approximate distribution. This might be the reason for the superior performance of approximate Bayesian bandit algorithms that consider only the KL divergence in practice.
Remark A.3 (Independent Prior). In multi-armed bandit problems, the prior distribution of mean rewards  $\Pi_0$  is typically chosen to be independent among each action  $\Pi_{0,1},\dots,\Pi_{0,K}$ , as this is the most reasonable and natural way without further information [1, 21, 20, 30]. In this case, the posteriors must also be independent among each action because of the Bayesian update (2). Therefore, our study focuses on the inference error on the distribution of each action in (3), which appears to be more realistic than the inference error on the joint distribution of all actions assumed in [34].
Appendix B Proofs of Results in Section 3.2