SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} A _ {t} := q _ {t} ^ {2} \big (\sigma_ {p, t} (1) ^ {2} + \sigma_ {p, t} (0) ^ {2} \big) - \big (\mu_ {p, t} (1) - \mu_ {p, t} (0) \big) ^ {2} \\ B _ {t} := q _ {t} ^ {2} \left(\sigma (1) ^ {2} + \sigma (0) ^ {2} + 2 \sigma_ {p, t} (0) ^ {2} M _ {t - 1} ^ {(1)}\right) \\ + 2 \left(b _ {t} - S _ {t - 1} ^ {\mathcal {T}} (1) + \mu_ {p, t} (0) M _ {t - 1} ^ {(1)}\right) \left(\mu_ {p, t} (1) - \mu_ {p, t} (0)\right) \tag {12} \\ \end{array}
$$
$$
C _ {t} := q _ {t} ^ {2} \sigma_ {p, t} (0) ^ {2} \left(M _ {t - 1} ^ {(1)}\right) ^ {2} + q _ {t} ^ {2} \sigma (0) ^ {2} M _ {t - 1} ^ {(1)} - \left(b _ {t} - S _ {t - 1} ^ {\mathcal {T}} (1) + \mu_ {p, t} (0) M _ {t - 1} ^ {(1)}\right) ^ {2}.
$$
Algorithm 1 finds the floor transform of the solutions of this equation and chooses  $m_t$  as the largest, positive integer that satisfies (11). If such a solution cannot be found, then either we do not have enough budget or the cost of the experiment is negligible (this accrues when  $\mu_{\mathrm{true}}(1) - \mu_{\mathrm{true}}(0) \gg 0$ , and the inequality in (11) will be strict for any choice of  $m_t$ . In the former case, Algorithm 1 sets  $m_t = 0$ ; in the latter case, it sets  $m_t = \lfloor N_t / 2 \rfloor$ . Therefore, by construction, the sequence  $(m_t)_{t \geq 1}$  output by Algorithm 1 guarantees that (1) and (2) hold, thereby defining a  $(\delta, B)$ -RRC experiment. Note that this approach directly solves for  $m_t$  from the quadratic equation  $A_t m_t^2 + B_t m_t + C_t = 0$ , bypassing the challenging task of estimating tail probabilities for potential choices of  $m_t$  through Monte-Carlo methods. By Algorithm 1, we can also conduct posterior inference on treatment effect after stage  $t$  using  $\mu_{p,t+1}(w), \sigma_{p,t+1}(w), w = 0, 1$  and estimate the remaining budget by  $B - \sum_{r=1}^{t} m_r (\mu_{p,r+1}(1) - \mu_{p,r+1}(0))$ .
Theorem 3.2. Assume the outcomes  $(Y_{i,t}(0),Y_{i,t}(1))_{i,t}$  are generated as in Definition 3.1. The experiment by Algorithm 1 is  $(\delta ,B)$ -RRC.
Even though our algorithms is derived from the conjugate Gaussian model we have found that it remains effective for broader outcome models. This is because the learning occurs essentially through computation of the first and second moments of past outcomes as in (5), (6), and Algorithm 1 tends to be successful so long as they are predictive of the outcome moments in future stages. The risk of ruin control remains approximately valid due to the law of large numbers and standard central limit theorem under specific conditions; see next section.
Finally, in Algorithm 1, the assumption is made that the population size  $N_{t}$  for the next stage is known to ensure that  $m_{t}$  does not exceed  $\lfloor N_t / 2\rfloor$ . In practice, we recommend estimating  $N_{t}$  and using the model output  $m_{t}$  to calculate the assignment probability  $p_t = m_t / N_t$ , the allows the experimenter to assign each incoming user to the treatment group with a probability of  $p_t$ .
Algorithm 1 Output ramp size adaptively
Input:  $B < 0, \delta \in [0,1)$   
1: Initialize  $t \gets 1, \prod_{r=1}^{0}(1 - \Delta_r) \gets 1$   
2: while  $\prod_{r=1}^{t-1}(1 - \Delta_r) > 1 - \delta$  do  
3: choose  $\Delta_t \in \left[0, \frac{1 - \delta}{\prod_{r=1}^{t-1}(1 - \Delta_r)} - 1\right], b_t \geq B$   
4: estimate data variance  $\sigma(w)^2, w = 0, 1$  (if unknown) using (6)  
5: compute  $\mu_{p,t}(w), \sigma_{p,t}^2(w), w = 0, 1$  by (5b), (5c), (4) and  $q_t, A_t, B_t, C_t$  by (11), (12)  
6: if  $\frac{b_t - S_{t-1}^T(1) - \tilde{\mu}_t(\lfloor N_t / 2 \rfloor)}{\sqrt{\tilde{\sigma}_t^2(\lfloor N_t / 2 \rfloor)}} \leq q_t$  then  $m_t \gets \lfloor N_t / 2 \rfloor$   
7: else if  $B_t^2 - 4A_tC_t < 0$  then  $m_t \gets 0$   
8: else  
9:  $\mathcal{M}_t \gets \left\{\left\lfloor \frac{-B_t + \sqrt{B_t^2 - 4A_tC_t}}{2A_t}\right\rfloor, \left\lfloor \frac{-B_t - \sqrt{B_t^2 - 4A_tC_t}}{2A_t}\right\rfloor\right\}$   
10:  $\mathcal{V}_t \gets \left\{m \in \mathcal{M}_t \cap [0, \frac{N_t}{2}]: \frac{b_t - S_{t-1}^T(1) - \tilde{\mu}_t(m)}{\tilde{\sigma}_t(m)} \leq q_t\right\}$   
11: if  $\mathcal{V}_t \neq \emptyset$  then  
12:  $m_t \gets \max V_t$   
13: else  
14:  $m_t \gets 0$   
15: end if  
16: end if  
17: Output  $m_t$ , conduct stage  $t$ -experiment and observe outcomes  $s_t^T(1), s_t^C(0)$   
18: compute  $M_t(0), M_t(1), S_t^T(1), S_t^C(0)$  by (5)  
19: update  $t \gets t + 1$   
20: end while
3.4 Robustness to non-identically distributed and non-Gaussian outcomes
We now derive conditions for the validity of Algorithm 1 under the assumption that experiment outcomes are independent.
Definition 3.4. The experiment outcomes  $(Y_{i,t}(0), Y_{i,t}(1))$  are independent across different units  $i$  and experiment stage  $t$ .
Definition 3.4 allows  $Y_{i,t}(0)$  and  $Y_{i,t}(1)$  to be dependent and/or discrete-valued (e.g., binary outcomes). In addition, the outcome distribution  $(Y_{i,t}(0), Y_{i,t}(1))$  can differ across  $i, t$ ; for instance, treatment effect may be non-stationary. The validity of Algorithm 1 under Definition 3.4 is now given in Theorem 3.3; we defer the proof to Appendix D.
Theorem 3.3. Assume the outcomes  $(Y_{i,t}(0),Y_{i,t}(1))_{i,t}$  satisfy Definition 3.4. The experiment by Algorithm 1 is  $(\delta ,B)$ -RRC if, for each stage  $t\geq 1$  where  $m_t\neq 0$ , the following conditions hold
$$
\mathbb {P} \left(\frac {s _ {t} ^ {\mathcal {T}} (1) - S _ {t} ^ {\mathcal {T}} (0) - \breve {\mu} _ {t}}{\breve {\sigma} _ {t}} \leq z _ {t} \mid \mathcal {F} _ {t - 1}\right) \leq \Phi (z _ {t}), \quad z _ {t} \leq \frac {\breve {\mu} _ {t} - \tilde {\mu} _ {t}}{\tilde {\sigma} _ {t} - \breve {\sigma} _ {t}}, \tag {13}
$$
$$
\breve {\mu} _ {t} := \mathbb {E} \left[ s _ {t} ^ {\mathcal {T}} (1) - S _ {t} ^ {\mathcal {T}} (0) \mid \mathcal {F} _ {t - 1} \right], \breve {\sigma} _ {t} ^ {2} := \mathbb {V} \left[ s _ {t} ^ {\mathcal {T}} (1) - S _ {t} ^ {\mathcal {T}} (0) \mid \mathcal {F} _ {t - 1} \right], z _ {t} := \frac {b _ {t} - S _ {t - 1} ^ {\mathcal {T}} (1) - \tilde {\mu} _ {t}}{\tilde {\sigma} _ {t}},
$$
and  $\tilde{\mu}_t\coloneqq \tilde{\mu}_t(m_t),\tilde{\sigma}_t\coloneqq \tilde{\sigma}_t(m_t)$  are defined by (10a), (10b), and  $s_t^T (1),S_t^T (0)$  are defined in (5).
We expect the first condition in (13) to hold as a consequence of central limit theorem for independent but non-identical random variables. Suppose  $\Delta_t \leq 0.5$ ,  $\forall t$  (i.e.,  $z_t \leq 0$ ). By law of large number for independent but non-identical random variables, the second condition in (13) holds if (i) we have chosen prior and model parameters conservatively such that
$$
\mu_ {0} (1) - \mu_ {0} (0) \leq \frac {1}{m _ {t}} \sum_ {i \in \mathcal {T} _ {1}} \mathbb {E} \left(Y _ {i, 1} (1) - Y _ {i, 1} (0)\right)
$$
$$
\sigma (0) ^ {2} + \sigma (1) ^ {2} + m _ {t} \cdot \left(\sigma_ {0} (1) ^ {2} + \sigma_ {0} (0) ^ {2}\right) \geq \frac {1}{m _ {t}} \sum_ {i \in \mathcal {T} _ {t}} \mathbb {V} \left(Y _ {i, t} (1) - Y _ {i, t} (0)\right)
$$
and (ii) if the treatment effects increase or stay roughly constant throughout the experiments
$$
\frac {1}{m _ {t}} \sum_ {i \in \mathcal {T} _ {t}} \mathbb {E} \left(Y _ {i, t} (1) - Y _ {i, t} (0)\right) \geq \frac {1}{M _ {t - 1} ^ {(1)}} \sum_ {r \in [ t - 1 ]} \sum_ {i \in \mathcal {T} _ {r}} \mathbb {E} \left[ Y _ {i, t} (1) - Y _ {i, t} (0) \right]
$$
and our variance estimates  $\sigma(0)^2, \sigma(1)^2$  are accurate or conservative in the sense that
$$
\sigma (0) ^ {2} \geq \frac {1}{M _ {t - 1} ^ {(1)}} \sum_ {r \in [ t - 1 ]} \sum_ {i \in \mathcal {T} _ {r}} \mathbb {V} \left[ Y _ {i, t} (0) \right], \sigma (0) ^ {2} + \sigma (1) ^ {2} \geq \frac {1}{m _ {t}} \sum_ {i \in \mathcal {T} _ {t}} \mathbb {V} \left(Y _ {i, t} (1) - Y _ {i, t} (0)\right).
$$
In summary, under Definition 3.4, the validity of Algorithm 1 depends on the accuracy and conservatism of the model's estimates based on past stages for the true treatment effect and volatility in the next stage. The algorithm's effectiveness may be compromised when there is a sudden decrease in treatment effect or a surge in outcome volatility in the next stage; see discussion in Appendix D.
4 Numerical and empirical experiments
Simulated ramp schedule We now examine the following three experimental scenarios, for each  $(Y_{i,t}(1),Y_{i,t}(0))_{i,t}$  are iid sampled from (3) with variance  $\sigma (1)^2 = \sigma (0)^2 = 10$  and means given below:
i) PTE: Positive treatment effect, with  $\mu_{\mathrm{true}}(0) = 0$ ,  $\mu_{\mathrm{true}}(1) = 1$
ii) NTE: Negative treatment effect, with  $\mu_{\mathrm{true}}(0) = 1$  and  $\mu_{\mathrm{true}}(1) = 0$
iii) PNTE: Negative to positive treatment effect, with  $\mu_{\mathrm{true}}(1)(t) = \min (-2 + 0.5(t - 1),2)$ .
For each scenario, we set  $T = 10$  with  $N_{t} = 500$ ,  $\forall t$  and we choose non-informative prior  $\mu_0(w) = 0$ ,  $\sigma_0(w)^2 = 100$ ,  $w = 0, 1$ . We assume model variance is known; however, using (6) to estimate the variance gives similar results. We repeat each scenario 500 times. Figure 1 (a)-(c) show median,  $25\%$  and  $75\%$  quantile of the simulated ramp schedules  $(m_t)_{t \in [T]}$  and  $(g),(h)$  show the budget surpluses  $\sum_{r \in [t]} \sum_{i \in \mathcal{T}_r} (Y_{i,t}(1) - Y_{i,t}(0)) - B$  produced given different choices of  $B$ ,  $\delta$ ,  $(b_t)_{t \in [T]}$ ,  $(\Delta)_{t \in [T]}$ .
Across the various scenarios, our model gives a reasonable ramp schedule. Large  $B$ ,  $\delta$  typically leads to more treated units and faster ramp-up. For the NPTE scenario, inadequate budget and low ruin tolerance can result in a failure to ramp up to  $50\%$  ( $m_t = 250$ ). We also found that reserving the budget for later stages by decreasing  $b_t$  or  $\Delta_t$  in the initial stages leads to a faster ramp-up because more budget is available to support a swift increase when the treatment effect turns positive. This suggests that the experimenter may want to consider reserving some budget for later stages if the treatment effect  $\mu_{p,t}(1) - \mu_{p,t}(0)$  has not stabilized.
For PNTE scenario, we compare our method to a Thompson-sampling bandit with tuning parameters  $c$  and prior  $\mu_0(1) = -2$ ,  $\mu_0(0) = 0$ ,  $\sigma_0(0)^2 = \sigma_0(1)^2 = 0.05$  (see [27] and Appendix G for details). The prior is chosen so that the bandit can initialize conservatively depending on  $c$ . It can be seen in Figure 1 (e),(i) that the ramp schedule generated is rather sub-optimal and does not respect the budget. It also follows a rigid pattern where with small  $c$ , the ramp-up initializes too aggressively, and for large  $c$ , the ramp-up proceeds too conservatively. These results demonstrate that our approach significantly outperforms the main existing alternative.
Semi-real LinkedIn ramp schedule comparison Appendix F gives group-level statistics from a 6-stage phased release run at LinkedIn. Due to privacy constraints, the individual-level data is not available and is simulated from (4) using stage-wise  $\mu_{\mathrm{true}}(w), \sigma(w)^2, w = 0,1$  (both unobserved). The ramp-up schedules for different tuning parameters are shown in Figure 1(d). It is noteworthy that the ramp-up schedule employed by LinkedIn's data scientists, which was chosen without considering a specific budget, is roughly consistent with the budget-rationing schedule denoted as "ration-budget LinkedIn" in the caption. Our results suggest that deducing the budget and risk tolerance associated with an experiment retroactively using our method is possible. We also run the experiment using Thompson sampling Bayesian bandit with the same prior as for NPTE above. In Figure 1(f), we again observe the rigidity issue: with small  $c$ , the ramp-up initializes too aggressively, and for large  $c$ , the ramp-up proceeds too conservatively.
(a) Ramp Schedule, PTE
(b) Ramp Schedule, NTE
(c) Ramp Schedule, NPTE
(d) Ramp schedule, LinkedIn
(e) Ramp Schedule, NPTE, TOM
(f) Ramp schedule, LinkedIn, TOM
(g) Budget surplus, NTE
(j)nor:1.22%/5%
Figure 1: Line plots (a)—(i) show the median,  $25\%$ ,  $75\%$  quantiles of either the treatment group sizes or the budget surplus for the 500 simulations of the different experiment setups (PTE, NTE, NPTE, LinkedIn) using our model and Thompson-sampling Bayesian bandit. Under the legends “ $(B,\delta)$ ”, we set  $b_{t} = B$ ,  $\Delta_{t} = 1 - (1 - \delta)^{1 / T}$ ,  $\forall t$ . We also use (i) “ration budget” to denote  $(B,\delta) = (-500,0.01)$ ,  $b_{t} = -400$ ,  $\forall t \leq 5$ ,  $b_{t} = -500$ ,  $\forall t > 5$  and  $\Delta_{t} = 1 - (1 - \delta)^{1 / T}$ ,  $\forall t$ ; (ii) “ration tolerance” to denote  $(B,\delta) = (-500,0.01)$ ,  $b_{t} = -500$ ,  $\forall t$  and  $\Delta_{t} = 0.0001$ ,  $\forall t \leq 5$ ,  $\Delta_{t} = 0.0019$ ,  $\forall t > 5$  (iii) “actual LinkedIn” to denote the actual ramp up schedule used by LinkedIn data scientists (iv) “ration-budget LinkedIn” to denote  $(B,\delta) = (-1500,0.01)$ ,  $b_{t} = -400$ ,  $t \geq 4$ ,  $b_{t} = -1500$ ,  $t > 4$ ,  $\Delta_{t} = 1 - (1 - \delta)^{1 / T}$ . Particularly, (e), (f), (i) are results using Thompson-sampling Bayesian bandit with different values of tuning parameter  $c$ , denoted by “TOM” in sub-caption, for experiment NPTE, LinkedIn. We set  $B = -500$  to produce (h), although the model is not budget-aware. Histograms (j)—(n) show distribution of the budget used over 5000 simulations. The red dashed line marks the budget available  $B = -500$ . “ $x\% / 5\%$ ” in the sub-captions denotes that the actual risk of ruin is  $x\%$  and the ruin tolerance is  $\delta = 5\%$ .
(h) Budget surplus, NPTE
(k) corr:1.52%/5%
(1)bern:  $1.30\% /5\%$
(i) Budget surplus, NPTE, TOM
(m) fat:  $1.24\% / 5\%$
(n) dec:18.28%/5%
Budget-spent distribution To explore how our algorithms controls the risk of ruin and budget spending, we simulate following experiments for 5,000 times and plot distribution of the budget spent  $\sum_{r\in [t]}\sum_{i\in \mathcal{T}_r}(Y_{i,t}(1) - Y_{i,t}(0))$  in Figure 1 (j)—(n): (i)norm:  $(Y_{i,t}(1),Y_{i,t}(0))_{i,t}$  are sampled iid from (3) with  $\mu_{\mathrm{true}}(0) = 0, \mu_{\mathrm{true}}(1) = 1, \sigma(0)^2 = \sigma(1)^2 = 10$ ; (ii) corr: same as norm except that for each  $i, t, Y_{i,t}(1)$  is correlated with  $Y_{i,t}(0)$  with correlation coefficient 0.8 (iii) bern:  $Y_{i,t}(0) \stackrel{\mathrm{iid}}{\sim} 6.4\mathsf{Bern}(p = 0.5786)$  and  $Y_{i,t}(1) \stackrel{\mathrm{iid}}{\sim} 6.4\mathsf{Bern}(p = 0.4224)$ ; (iv) fat:  $Y_{i,t}(0) \stackrel{\mathrm{iid}}{\sim} 1 + \sqrt{5}\mathsf{t}_4$ , and  $Y_{i,t}(1) \stackrel{\mathrm{iid}}{\sim} \sqrt{5}t_4^2$  (v) dec: same as norm except  $\mu_{\mathrm{true}}(1)(t) = -(t - 1)$ . Note that (iii), (iv) is configured so that  $\mathbb{E}[Y_{i,t}(1) - Y_{i,t}(0)] = 1, \mathbb{V}(Y_{i,t}(0)) = \mathbb{V}(Y_{i,t}(1)) = 10$ . For all the above experiments, we run  $T = 10$  stages with  $N_t = 500, \forall t$  and we use non-informative prior  $\mu_0(w) = 0, \sigma_0(w)^2 = 100, w = 0, 1$ .
As shown in Figure 1 (j)-(m) the model successfully controls risk of ruin for (i)-(iv). The actual ruin risk is at a reasonable level ( $\sim 1.2\%$ ) compared to the ruin tolerance given ( $5\%$ ). Note that the actual ruin risk are close for different outcome distribution. This is a consequence of central limit theorem and law of large numbers as discussed in Section 3.4. The model fails to control risk of ruin for (v) as expected since the treatment effect keeps decreasing and the model assigns treatment based on past stages which leads to higher-than-expected costs (cf. Section 3.4).
2Where  $t_4$  is a Student-t distribution with 4 degrees of freedom.
[1] Apple. App store: Release a version update in phases, 2023.
[2] Maryam Aziz, Emilie Kaufmann, and Marie-Karelle Riviere. On multi-armed bandit designs for dose-finding clinical trials. The Journal of Machine Learning Research, 22(1):686-723, 2021.
[3] Eytan Bakshy, Dean Eckles, and Michael S Bernstein. Designing and deploying online field experiments. In Proceedings of the 23rd international conference on World wide web, pages 283-292, 2014.
[4] Donald A Berry. Modified two-armed bandit strategies for certain clinical trials. Journal of the American Statistical Association, 73(362):339–345, 1978.
[5] Iavor Bojinov and Somit Gupta. Online experimentation: Benefits, operational and methodological challenges, and scaling guide. Harvard Data Science Review, 4(3), 2022.
[6] Iavor Bojinov and Karim R. Lakhani. Experimentation at yelp. Harvard Business School Case 621-064, 2020.
[7] Iavor Bojinov and Neil Shephard. Time series experiments and causal estimands: Exact randomization tests and trading. Journal of the American Statistical Association, page Forthcoming, 2019.
[8] Asaf Cassel, Shie Mannor, and Assaf Zeevi. A general approach to multi-armed bandits under risk criteria. In Conference on learning theory, pages 1295–1306. PMLR, 2018.
[9] Semih Cayci, Atilla Eryilmaz, and Rayadurgam Srikant. Budget-constrained bandits over general cost and reward distributions. In International Conference on Artificial Intelligence and Statistics, pages 4388-4398. PMLR, 2020.
[10] Joel QL Chang and Vincent YF Tan. A unifying theory of thompson sampling for continuous risk-averse bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6159–6166, 2022.
[11] David. R. Cox. Planning of Experiments. John Wiley & Sons, New York, 1958.
[12] Debojit Das, Shweta Jain, and Sujit Gujar. Budgeted combinatorial multi-armed bandits. arXiv preprint arXiv:2202.03704, 2022.
[13] WF Eberlein. On euler's infinite product for the sine. Journal of Mathematical Analysis and Applications, 58(1):147-151, 1977.
[14] Nicolas Galichet, Michele Sebag, and Olivier Teytaud. Exploration vs exploitation vs safety: Risk-aware multi-armed bandits. In *Asian Conference on Machine Learning*, pages 245–260. PMLR, 2013.
[15] Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesian data analysis. Chapman and Hall/CRC, 1995.
[16] Google. Play console help: Release app updates with staged rollouts, 2023.
[17] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education, 2010.
[18] Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1168-1176, 2013.
[19] Odalric-Ambrym Maillard. Robust risk-averse stochastic multi-armed bandits. In Algorithmic Learning Theory: 24th International Conference, ALT 2013, Singapore, October 6-9, 2013. Proceedings 24, pages 218-233. Springer, 2013.
[20] Jialiang Mao and Iavor Bojinov. Quantifying the value of iterative experimentation. arXiv preprint arXiv:2111.02334, 2021.
[21] Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. Batched bandit problems. The Annals of Statistics, pages 660–681, 2016.
[22] Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
[23] Amir Sani, Alessandro Lazaric, and Rémi Munos. Risk-aversion in multi-armed bandits. Advances in neural information processing systems, 25, 2012.
[24] Sama Shrestha and Sonia Jain. A bayesian-bandit adaptive design for n-of-1 clinical trials. Statistics in Medicine, 40(7):1825–1844, 2021.
[25] Jerzy Splawa-Neyman, Dorota M Dabrowska, and TP Speed. On the application of probability theory to agricultural experiments. essay on principles. section 9. Statistical Science, pages 465-472, 1990.
[26] Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 17-26, 2010.
[27] Peter F Thall and J Kyle Wathen. Practical bayesian adaptive randomisation in clinical trials. European Journal of Cancer, 43(5):859-866, 2007.
[28] Sattar Vakili and Qing Zhao. Risk-averse multi-armed bandit problems under mean-variance measure. IEEE Journal of Selected Topics in Signal Processing, 10(6):1093-1111, 2016.
[29] Ryo Watanabe, Junpei Komiyama, Atsuyoshi Nakamura, and Mineichi Kudo. Kl-ucb-based policy for budgeted multi-armed bandits with stochastic action costs. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 100(11):2470–2486, 2017.
[30] Tong Xia, Sumit Bhardwaj, Pavel Dmitriev, and Aleksander Fabijan. Safe velocity: a practical guide to software deployment at scale using controlled rollout. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 11–20. IEEE, 2019.
[31] Yingce Xia, Wenkui Ding, Xu-Dong Zhang, Nenghai Yu, and Tao Qin. Budgeted bandit problems with continuous random costs. In Asian conference on machine learning, pages 317-332. PMLR, 2016.
[32] Yingce Xia, Haifang Li, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Thompson sampling for budgeted multi-armed bandits. arXiv preprint arXiv:1505.00146, 2015.
[33] Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. From infrastructure to culture: A/b testing challenges in large scale social networks. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2227-2236, 2015.
[34] Ya Xu, Weitao Duan, and Shaochen Huang. Sqr: balancing speed, quality and risk in online experiments. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 895-904, 2018.
[35] Qiuyu Zhu and Vincent Tan. Thompson sampling algorithms for mean-variance bandits. In International Conference on Machine Learning, pages 11599-11608. PMLR, 2020.
A Decompose risk-of-ruin to individual stages
Proof of Theorem 3.1. The last claim is trivial: it is easy to verify that (1) and (2) hold if we let  $\mathcal{T}_t = \emptyset$  for all  $t\in [T]$ . Now we prove the first and the second claim. The case for  $T = 1$  is trivial. We assume  $T\geq 2$ . For any  $t = 2,\dots ,T$ , we have that
$$
\begin{array}{l} \mathbb {P} \left(R _ {t} \leq B\right) = \mathbb {E} \left[ \mathbb {P} \left(R _ {t} \leq B \mid \mathcal {F} _ {t - 1}\right) \right] \\ = \mathbb {E} \left[ \mathbb {P} \left(R _ {t} \leq B, R _ {t - 1} \leq B \mid \mathcal {F} _ {t - 1}\right) + \mathbb {P} \left(R _ {t} \leq B, R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) \right] \\ \stackrel {(a)} {\leq} \mathbb {P} \left(R _ {t - 1} \leq B\right) + \mathbb {E} \left[ \mathbb {P} \left(R _ {t} \leq B, R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) \right] \\ = \mathbb {P} \left(R _ {t - 1} \leq B\right) + \mathbb {E} \left[ \mathbb {I} \left(\mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) > 0\right) \right. \\ \times \mathbb {P} \left(R _ {t} \leq B \mid R _ {t - 1} > B, \mathcal {F} _ {t - 1}\right) \mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) \\ \left. + \mathbb {I} \left(\mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) = 0\right) \mathbb {P} \left(R _ {t} \leq B, R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {=} \mathbb {P} \left(R _ {t - 1} \leq B\right) + \mathbb {E} \left[ \mathbb {I} \left(\mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1} > 0\right)\right) \right. \\ \left. \times \mathbb {P} \left(R _ {t} \leq B \mid R _ {t - 1} > B, \mathcal {F} _ {t - 1}\right) \mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(c)} {\leq} \mathbb {P} \left(R _ {t - 1} \leq B\right) + \Delta_ {t} \mathbb {E} \left[ \mathbb {I} \left(\mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) > 0\right) \mathbb {P} \left(R _ {t - 1} > B \mid \mathcal {F} _ {t - 1}\right) \right] \\ = \mathbb {P} \left(R _ {t - 1} \leq B\right) + \Delta_ {t} \mathbb {P} \left(R _ {t - 1} > B\right) \\ \end{array}
$$