Title: Balancing Risk and Reward: A Batched-Bandit Strategy for Automated Phased Release
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandit algorithms

online learning

causality

Bayesian inference
Abstract: Phased releases are a common strategy in the technology industry for gradually releasing new products or updates through a sequence of A/B tests in which the number of treated units gradually grows until full deployment or deprecation. Performing phased releases in a principled way requires selecting the proportion of units assigned to the new release in a way that balances the risk of an adverse effect with the need to iterate and learn from the experiment rapidly. In this paper, we formalize this problem and propose an algorithm that automatically determines the release percentage at each stage in the schedule, balancing the need to control risk while maximizing ramp-up speed. Our framework models the challenge as a constrained batched bandit problem that ensures that our pre-specified experimental budget is not depleted with high probability. Our proposed algorithm leverages an adaptive Bayesian approach in which the maximal number of units assigned to the treatment is determined by the posterior distribution, ensuring that the probability of depleting the remaining budget is low. Notably, our approach analytically solves the ramp sizes by inverting probability bounds, eliminating the need for challenging rare-event Monte Carlo simulation. It only requires computing means and variances of outcome subsets, making it highly efficient and parallelizable.

Balance Risk and Reward: A Batched-Bandit Strategy for Automated Phased Release

Abstract

Phased releases are a common strategy in the technology industry for gradually releasing new products or updates through a sequence of A/B tests in which the number of treated units gradually grows until full deployment or deprecation. Performing phased releases in a principled way requires selecting the proportion of units assigned to the new release in a way that balances the risk of an adverse effect with the need to iterate and learn from the experiment rapidly. In this paper, we formalize this problem and propose an algorithm that automatically determines the release percentage at each stage in the schedule, balancing the need to control risk while maximizing ramp-up speed. Our framework models the challenge as a constrained batched bandit problem that ensures that our pre-specified experimental budget is not depleted with high probability. Our proposed algorithm leverages an adaptive Bayesian approach in which the maximal number of units assigned to the treatment is determined by the posterior distribution, ensuring that the probability of depleting the remaining budget is low. Notably, our approach analytically solves the ramp sizes by inverting probability bounds, eliminating the need for challenging rare-event Monte Carlo simulation. It only requires computing means and variances of outcome subsets, making it highly efficient and parallelizable.

1 Introduction

Phased release, also known as staged rollout, is a widely used strategy in the technology industry that involves gradually releasing a new product or update to larger audiences over time [17, 30]. For example, Apple's App Store offers a phased release option where application updates are released over a 7-day period on a fixed schedule [1]. Google Play Console provides a similar feature with more flexibility in the release schedule [16]. Typically, the audiences are randomly selected at each stage from the set of all customers, and so phased releases can be thought of as a sequence of A/B tests (or randomized experiments) in which the proportion of units assigned to the treatment group changes until either the product or update is fully launched or deprecated [26, 18, 3, 33, 6]. The process of combining phased releases with A/B tests is often called controlled rollout or iterative experiments and provides companies with an important mechanism to gather feedback on early product versions [30, 20, 5].

The key advantage of phased release is its ability to mitigate risks associated with launching a new product or update directly to all users. The potential impact of faulty features is limited by releasing the update first to a small percentage of the users (i.e., the treatment group). However, this risk-averse approach introduces an opportunity cost for slowly launching beneficial features, which quickly adds up for companies that release thousands of features yearly [34]. Therefore, when designing a phased release schedule, it is important to determine the release percentage (known as ramp schedule) at each stage that balances the need to control risk while maximizing the speed of ramp-up. ' This paper proposes an algorithm to address this challenge by automatically determining the release

percentage for the next phase based on observations from previous stages. Specifically, we frame the challenge as a budget-constrained batched bandit problem. For each batch, we aim to determine the assignment probabilities of newly arrived users while keeping the probability of depleting a pre-specified experimental budget, where the experiment's cost is the cumulative treatment effect that are not directly observed. Formally, we derive recursive relations that decompose the risk of ruin (depleting the budget) of a phased release to the individual stages in the sense that the risk of ruin of the entire experiment is controlled if stage-wise ruin probabilities are controlled. Our algorithm is Bayesian in the sense that it learns from past observations by computing the posteriors of a conjugate Gaussian model and uses these parameters to infer the remaining budget and other cost-related quantities. However, the algorithm is robust to misspecifications and works well even when underlying outcomes are far from Gaussian by law of large number and central limit theorem; nevertheless, in Appendix E, we provide an extension to non-Gaussian outcomes. Finally, the next stage's assignment probabilities are derived from the posterior distribution and the stage-wise risk tolerances. Notably, our approach solves ramp sizes analytically from inverting the ruin probability upper bounds, avoiding challenging rare-event Monte-Carlo simulation for budget depletion events and data imputation procedures for unobserved counterfactual outcomes.

1.1 Literature review

While many firms have guidelines on how to conduct a phased release process, these guidelines are often ad-hoc and qualitative, making it difficult to create executable ramp schedules. The SQR framework in [34] is the first attempt to address this problem by providing quantitative guidance. Our work differs significantly from SQR. Our algorithm adopts a fully Bayesian approach, enabling us to incorporate prior information on the risk of a feature in a probabilistic manner when initiating a ramp. Additionally, unlike SQR, our approach introduces a 'shared budget' over the entire phased release, allowing the budget to be sequentially adjusted based on the observations from prior iterations. Finally, our algorithm is robust to modifications made to the treatment during experiments and different outcome models.

Our work is notably distinct from the risk-averse multiarmed bandit approaches considered in previous research [19, 14, 35, 28, 23, 10, 8]. In these approaches, the agent considers the expected variability in expected rewards to identify and avoid less predictable (and therefore risky) actions, without considering a budget constraint. A related literature focuses on batched and Bayesian variants of these methods in multi-stage clinical trials [4, 27, 21, 24, 2]. While this literature also aims to determine treatment assignment for each stage of the experiment, it differs from our setting in two key aspects: (i) the objective is to maximize treatment effect while balancing exploration of treatment arms, rather than rapidly ramping up experiments, and (ii) to the best of our knowledge, no clinical trials paper has addressed the imposition of a budget for potential adverse treatment effects. Hence, bandit approaches developed for clinical trials cannot be directly applied to our setting. To illustrate the difference, we present a numerical simulation of a Thompson sampling-based Bayesian bandit from [27] and highlight that budget spent and the aggressiveness of the ramp-up schedule depends on model tuning in a very unpredictable way, making the ramp-up schedule far from ideal. Another related literature is budgeted multiarmed bandits [32, 31, 9, 29, 12]. However, most budgeted bandit algorithms are developed for settings very different from ours and do not consider risk-of-ruin control or handle unobserved costs. Therefore, these algorithms cannot be directly applied to our specific scenario.

Notation. Let N be the set of non-negative integers and N + := N \ { 0 } . Let R , R + denote the set of real numbers and positive real numbers respectively. [ N ] := 1, . . . , N for N ∈ N + . σ ( · ) is the generated σ -algebra. X ∈ F if random variable X is measurable to F . [ X | F ] denotes a random variable with distribution P ( X ∈ · | F ) for a random variable X and σ -algebra F .

2 Risk-of-ruin-constrained experiment and strategy overview

2.1 Risk-of-ruin-constrained experiment

Consider a scenario in which a single feature is released to a sequence of subpopulations N t consisting of N t units at stages t = 1, ..., T , where T is not necessarily fixed. At each stage, we randomly assign treatment to a group of units denoting the indexing set T t , while the control group with indexing

set C t . The size of the treatment group at stage t is |T t | = m t and the size of the control group is |C t | = N T -m t .

Our paper adopts the Neyman-Rubin framework for causal inference [22, 25, 7], where the potential outcome of each unit i ∈ N t during experiment stage t under control and treatment are denoted by Y i , t (0) ∈ R and Y i , t (1) ∈ R , respectively 1 . Appendix E provides the extension to multivariate outcomes. The treatment assignment of unit i ∈ N t at stage t is denoted by W i , t . Since each unit only receives a single treatment at each stage, we only observe Y i , t ( W i , t ) during the experiment, not the counterfactual Y i , t (1 -W i , t ) (we are explicitly assuming that there is full compliance).

Let F t = σ ( ( Y T k , k (1)) k ∈ [ t ] , ( Y C k , k (0)) k ∈ [ t ] , ( W i , k ) i ∈N , k ∈ [ t ] ) be the σ -algebra generated by the treatment assignment and the observed experiment outcome in the first t stages, with F 0 representing the trivial σ -algebra. In our setting, the experimenter aims to ramp up the experiment to the "maxpower stage" (50% of the population placed in treatment) as quickly as possible while avoiding the risk of a large negative business impact (or cost). We define the cost of the experiment as the treatment effect on the treated. See generalized cost in Appendix E.

Definition 2.1 (Experiment cost) . The cost of the experiment from stage t ∈ [ T ] is r t := ∑ i ∈T t Y i , t (1) -Y i , t (0) , where r t = 0 if T t = ∅ . The cumulative cost is R t := ∑ k ∈ [ t ] r k .

Throughout, r t < 0 corresponds to a negative business impact; our goal is to control the experiment cost by setting a budget B < 0 and imposing the cost constraint R T > B . Since the outcomes are stochastic, we require this cost constraint to be satisfied with probability at least 1 -δ for some δ ∈ [0, 1) set before the experiment. Our goal is then to adaptively determine the size of the treatment group based on the observed data while satisfying our risk constraint. We refer to such an experiment as a risk-of-ruin-constrained (RRC) experiment.

Definition 2.2 (RRC experiment) . Fix any B < 0, δ ∈ [0, 1) . A ( δ , B ) -RRC experiment running for T stages selects the size of T t before t -th stage of the experiment such that P ( R T > B ) ≥ 1 -δ .

2.2 Strategy overview

Our goal is to determine the number of users to assign to the treatment m t ∈ [0, N t / 2] , such that ∑ T i =1 m t is maximized while ensuring that the experiment is ( B , δ ) -RRC.

Our strategy is to decompose the overall constraint ( i.e. , that the experiment is ( B , δ ) -RRC) into a sequence of stage-wise adaptive constraints using Theorem 3.1, below. We then sequentially maximize m t under the stage-wise constraint for the t -th stage. Under the Gaussian model, the stage-wise constraints can be directly solved using a simple quadratic equation to obtain the maximum m t .

Although our algorithm solves a relaxed version of the original optimization problem, our solution has crucial practical implications. Theorem 3.1 decomposes the ( B , δ ) -RRC constraints in an adaptive fashion, i.e. , if the feature turns out to be safe, the stage-wise constraints will relax in response, and the experiment will ramp up quickly.

3 Model and the algorithm

3.1 Decompose the risk of ruin

Our experimental design is based on the following theorem, which identifies a sequence of sufficient conditions for a sequential experiment to be ( δ , B ) -RRC. We defer its proof to Appendix A.

Theorem 3.1. Fix B < 0 and δ ∈ [0, 1) . For any stopping time T ≥ 1 , let ( b t ) t ∈ [ T ] be a budget sequence, such that b t ( i ) ≥ B , ∀ t ∈ [ T ] , and (∆ t ) t ∈ [ T ] be a risk tolerance sequence, such that

∆ t ∈ [0, 1), ∀ t ∈ [ T ] and 1 -∏ T t =1 (1 -∆ t ) ( ii ) ≤ δ . Then, if ( T t ) t ∈ [ T ] is chosen such that for t = 1 ,

and for any t = 2, ..., T , almost surely,

then P ( R T > B ) ≥ 1 -δ . This inequality is tight when ( i ) -( iv ) are all equalities and r t ≤ 0, ∀ t ∈ [ T ] almost surely. Furthermore, if we set T t ←∅ , (1) , (2) always hold.

Recall, B < 0 denotes the budget and δ ∈ [0, 1) is our risk tolerance that controls the risk of ruin ( i.e., the probability of exceeding the budget); both need to be fixed a priori . In general, smaller δ leads to more conservative experimentation and slower releases. The sequence ( b t ) t ≥ 1 'rations' the budget: setting b t < B at stage t reserves B -b t budget for later stages, which may be beneficial when the released feature is expected to undergo modifications during the experiment [20]. To quickly scale the experiment, we can set b t = B , ∀ t . The sequence (∆ t ) t ≥ 1 distributes the overall tolerance δ to individual stages by δ = 1 -∏ T i -t (1 -∆ t ) allowing us to customize the tolerance for individual stages. If T is fixed a priori, we can uniformly distribute the tolerance by setting ∆ t = 1 -(1 -δ ) 1 /T , ∀ t ∈ [ T ] .

Theorem 3.1 breaks the risk constraint P ( R T ≤ B ) ≤ δ into stage-wise constraints in the form of (1),(2). The idea is to control current-stage cumulative experiment cost R t given past observations F t and determine the treatment assignment based on posterior inference of the remaining budget. In our setting, the goal is to maximize m t = |T t | subject to (1),(2). Note that the first line in (2) stops the experiment when the model estimates that the budget is exhausted, while the second line sets the stage cost r t below the remaining budget b t -R t -1 with high probability. We require assumptions on the data-generating model to derive an explicit algorithm, which we present in the next sections.

Generally, the total number of stages T , stage-wise budget and tolerance ( b t ) t ≥ 1 , (∆ t ) t ≥ 1 can be determined dynamically during the process. That is, we can define ∆ t and b t just before stage t as long as ∏ t r =1 (1 -∆ t ) ≤ 1 -δ and b t ≥ B . If we plan to terminate the experiment after stage t , we can define ∆ t such that ∏ t r =1 (1 -∆ r ) = 1 -δ and b t ≥ B is attained in which case T = t . If is also possible to have T = + ∞ and ∏ ∞ t =1 (1 -∆ t ) = 1 -δ . For example, choosing ∆ t = ( γ glyph[star] /t ) 2 where γ glyph[star] is the unique solution of sinc ( γ glyph[star] ) = 1 -δ on [0, 1] (cf. [13, Eq. (1)]) satisfy our condition.

Note that the decomposition scheme in Theorem 3.1 is formulated such that P ( R T ≤ B ) ≤ δ is tight if inequalities (i)-(iv) are tight. Practically, this means that the ramp-up schedules obtained through this approach are typically not overly conservative, unlike approaches that leverage a unionbound for risk decomposition. Finally, Theorem 3.1 holds for a general definition of the cost r t := r t ( ( Y i , t ) i ∈N t , T t ) such that r t = 0 if T t = ∅ , making it useful in other budgeted online problems beyond our setting.

3.2 Gaussian outcome model

For this subsection, make the following model assumptions on the outcomes distribution; appendix E provides the extension to general outcome models.

Definition 3.1 (Conjugate Gaussian outcomes) . Let the unknown model parameters µ true (0), µ true (1) ∈ R satisfy the prior µ true ( w ) ∼ N ( µ 0 ( w ), σ 0 ( w ) 2 ) for w = 0, 1 independently, where µ 0 (0), µ 0 (1) ∈ R , σ 0 (0) 2 , σ 0 (1) 2 ∈ R + are hyperparameters. The experiment outcome of unit i at stage t are distributed independently and identically as

where σ (0) 2 , σ (1) 2 ∈ R + are hyperparameters.

The unknown parameters µ true (0) and µ true (1) represent the intrinsic quality of the feature before and after the update, as measured by a specific metric. If µ true (1) -µ true (0) < 0 , the feature update is likely to have a negative business impact, i.e., Y i , t (1) -Y i , t (0) < 0 .

To derive the posterior distribution we need the following statistics. For t = 1, w = 0, 1

and for t ≥ 2, w = 0, 1

where M (1) t := ∑ r ∈ [ t ] m r and M (0) t := ∑ r ∈ [ t ] N r -m r are the cumulative number of users in the treatment and control groups up to stage t , respectively. In (5a), S T t ( w ) and S C t ( w ) represent the cumulative sum of outcomes for w = 0, 1 in the treatment and control groups up to stage t , while s T t ( w ) and s C t ( w ) represent the sum of outcomes at stage t . In equation (5b), µ p , t ( w ) represents the posterior mean of µ true ( w ) , while in equation (5c), σ p , t ( w ) 2 represents the posterior variance of µ true ( w ) , for w = 0, 1 . When lacking prior information, we suggest using a non-informative priors by setting µ 0 (0) = µ 0 (1) = 0 and σ 0 (0) 2 , σ 0 (1) 2 sufficiently large.

The model parameters σ (0) 2 and σ (1) 2 at stage t ≥ 2 can be estimated using unbiased and consistent estimators. For w = 0, 1 let

For t = 1 , some prior estimate can be used, either from a similar experiment or from a small-scale pretrial run.

3.3 An algorithm for the sample size in an RRC experiment

We now derive an explicit algorithm from Theorem 3.1 that to outputs ( m t ) t ≥ 1 , the treatment group size at stage t , such that, the experiment is ( δ , B ) -RRC. Recall that for an experiment to be ( δ , B ) -RRC, it suffices that (1), (2) holds for each t ≥ 1 . Under Definition 3.1, we have that (i) ( Y i , t ) i , t are exchangeable random variables (ii) for any t ≥ 2 , P ( S T t -1 (0) < S T t -1 (1) -B |F t -1 ) > 0 almost surely for any choice of m [ t -1] . Combining these observations, we get that (1), (2) hold if for each t ≥ 1 ,

Lemma 3.2 provides an upper bound of the left hand size of (7); the proof is in Appendix B.

Lemma 3.2 (Stochastic domination) . Assume the outcomes ( Y i , t (0), Y i , t (1)) i , t are generated as in Definition 3.1. For any ≥ 1 , almost surely,

Using Lemma 3.2, for (1) and (2) to hold, it suffices to choose any m t ∈ N such that

and set m t = 0 if such m t does not exist. From posterior-predictive formulas for the conjugate Gaussian model in Definition 3.1 (see (15e), (15f) in Appendix C), we have [ s T t (1) -S T t (0) ∣ ∣ ∣ ∣ F t -1 ] ∼

N (˜ µ t ( m t ), ˜ σ 2 t ( m t )), where

Combining the above with (9) yields the following Lemma.

Lemma 3.3. Assume the outcomes ( Y i , t (0), Y i , t (1)) i , t are generated as in Definition 3.1. For each t ≥ 1 , the inequality (9) holds if and only if

where Φ -1 denotes inverse CDF of the standard normal distribution.

Replace the inequality in (11) with equality and square both sides gives us the quadratic equation A t · m 2 t + B t · m t + C t = 0 where

Algorithm 1 finds the floor transform of the solutions of this equation and chooses m t as the largest, positive integer that satisfies (11). If such a solution cannot be found, then either we do not have enough budget or the cost of the experiment is negligible (this accrues when µ true (1) -µ true (0) glyph[greatermuch] 0 ), and the inequality in (11) will be strict for any choice of m t . In the former case, Algorithm 1 sets m t = 0 ; in the latter case, it sets m t = glyph[floorleft] N t / 2 glyph[floorright] . Therefore, by construction, the sequence ( m t ) t ≥ 1 output by Algorithm 1 guarantees that (1) and (2) hold, thereby defining a ( δ , B ) -RRC experiment. Note that this approach directly solves for m t from the quadratic equation A t m 2 t + B t m t + C t = 0 , bypassing the challenging task of estimating tail probabilities for potential choices of m t through Monte-Carlo methods. By Algorithm 1, we can also conduct posterior inference on treatment effect after stage t using µ p , t +1 ( w ), σ p , t +1 ( w ), w = 0,1 and estimate the remaining budget by B -∑ t r =1 m r ( µ p , r +1 (1) -µ p , r +1 (0)) .

Theorem 3.2. Assume the outcomes ( Y i , t (0), Y i , t (1)) i , t are generated as in Definition 3.1. The experiment by Algorithm 1 is ( δ , B ) -RRC.

Even though our algorithms is derived from the conjugate Gaussian model we have found that it remains effective for broader outcome models. This is because the learning occurs essentially through computation of the first and second moments of past outcomes as in (5), (6), and Algorithm 1 tends to be successful so long as they are predictive of the outcome moments in future stages. The risk of ruin control remains approximately valid due to the law of large numbers and standard central limit theorem under specific conditions; see next section.

Finally, in Algorithm 1, the assumption is made that the population size N t for the next stage is known to ensure that m t does not exceed glyph[floorleft] N t / 2 glyph[floorright] . In practice, we recommend estimating N t and using the model output m t to calculate the assignment probability p t = m t /N t , the allows the experimenter to assign each incoming user to the treatment group with a probability of p t .

Algorithm 1 Output ramp size adaptively

glyph[negationslash]

3.4 Robustness to non-identically distributed and non-Gaussian outcomes

We now derive conditions for the validity of Algorithm 1 under the assumption that experiment outcomes are independent.

Definition 3.4. The experiment outcomes ( Y i , t (0), Y i , t (1)) are independent across different units i and experiment stage t .

Definition 3.4 allows Y i , t (0) and Y i , t (1) to be dependent and/or discrete-valued ( e.g. , binary outcomes). In addition, the outcome distribution ( Y i , t (0), Y i , t (1)) can differ across i , t ; for instance, treatment effect may be non-stationary. The validity of Algorithm 1 under Definition 3.4 is now given in Theorem 3.3; we defer the proof to Appendix D.

glyph[negationslash]

Theorem 3.3. Assume the outcomes ( Y i , t (0), Y i , t (1)) i , t satisfy Definition 3.4. The experiment by Algorithm 1 is ( δ , B ) -RRC if, for each stage t ≥ 1 where m t = 0 , the following conditions hold

where

We expect the first condition in (13) to hold as a consequence of central limit theorem for independent but non-identical random variables. Suppose ∆ t ≤ 0.5, ∀ t ( i.e., z t ≤ 0 ). By law of large number for independent but non-identical random variables, the second condition in (13) holds if (i) we have chosen prior and model parameters conservatively such that

and (ii) if the treatment effects increase or stay roughly constant throughout the experiments

and our variance estimates σ (0) 2 , σ (1) 2 are accurate or conservative in the sense that

In summary, under Definition 3.4, the validity of Algorithm 1 depends on the accuracy and conservatism of the model's estimates based on past stages for the true treatment effect and volatility in the next stage. The algorithm's effectiveness may be compromised when there is a sudden decrease in treatment effect or a surge in outcome volatility in the next stage; see discussion in Appendix D.

4 Numerical and empirical experiments

Simulated ramp schedule We now examine the following three experimental scenarios, for each ( Y i , t (1), Y i , t (0)) i , t are iid sampled from (3) with variance σ (1) 2 = σ (0) 2 = 10 and means given below:

PTE : Positive treatment effect, with µ true (0) = 0 , µ true (1) = 1 ;

ii) NTE : Negative treatment effect, with µ true (0) = 1 and µ true (1) = 0 ;

iii) PNTE : Negative to positive treatment effect, with µ true (1)( t ) = min ( -2 + 0.5( t -1), 2) .

For each scenario, we set T = 10 with N t = 500, ∀ t and we choose non-informative prior µ 0 ( w ) = 0, σ 0 ( w ) 2 = 100, w = 0, 1 . We assume model variance is known; however, using (6) to estimate the variance gives similar results. We repeat each scenario 500 times. Figure 1 (a)-(c) show median, 25% and 75% quantile of the simulated ramp schedules ( m t ) t ∈ [ T ] and (g),(h) show the budget surpluses ∑ r ∈ [ t ] ∑ i ∈T r ( Y i , t (1) -Y i , t (0)) -B produced given different choices of B , δ , ( b t ) t ∈ [ T ] , (∆) t ∈ [ T ] .

Across the various scenarios, our model gives a reasonable ramp schedule. Large B , δ typically leads to more treated units and faster ramp-up. For the NPTE scenario, inadequate budget and low ruin tolerance can result in a failure to ramp up to 50% ( m t = 250 ). We also found that reserving the budget for later stages by decreasing b t or ∆ t in the initial stages leads to a faster ramp-up because more budget is available to support a swift increase when the treatment effect turns positive. This suggests that the experimenter may want to consider reserving some budget for later stages if the treatment effect µ p , t (1) -µ p , t (0) has not stabilized.

For PNTE scenario, we compare our method to a Thompson-sampling bandit with tuning parameters c and prior µ 0 (1) = -2, µ 0 (0) = 0, σ 0 (0) 2 = σ 0 (1) 2 = 0.05 (see [27] and Appendix G for details). The prior is chosen so that the bandit can initialize conservatively depending on c . It can be seen in Figure 1 (e),(i) that the ramp schedule generated is rather sub-optimal and does not respect the budget. It also follows a rigid pattern where with small c , the ramp-up initializes too aggressively, and for large c , the ramp-up proceeds too conservatively. These results demonstrate that our approach significantly outperforms the main existing alternative.

Semi-real LinkedIn ramp schedule comparison Appendix F gives group-level statistics from a 6-stage phased release run at LinkedIn. Due to privacy constraints, the individual-level data is not available and is simulated from (4) using stage-wise µ true ( w ), σ ( w ) 2 , w = 0, 1 (both unobserved). The ramp-up schedules for different tuning parameters are shown in Figure 1,(d). It is noteworthy that the ramp-up schedule employed by LinkedIn's data scientists, which was chosen without considering a specific budget, is roughly consistent with the budget-rationing schedule denoted as " ration-budget Linkedin " in the caption. Our results suggest that deducing the budget and risk tolerance associated with an experiment retroactively using our method is possible. We also run the experiment using Thompson sampling Bayesian bandit with the same prior as for NPTE above. In Figure 1(f), we again observe the rigidity issue: with small c , the ramp-up initializes too aggressively, and for large c , the ramp-up proceeds too conservatively.

6

PTE

NTE

NPTE

(d) Ramp schedule,

LinkedIn

NPTE, TOM

(f) Ramp schedule,

LinkedIn, TOM

(g) Budget surplus,

NTE

(h) Budget surplus,

NPTE

(i) Budget surplus,

NPTE, TOM

(j)

nor

:1.22%/5%

(k)

corr

:1.52%/5%

(l)

bern

:1.30%/5%

(m)

fat

:1.24%/5 %

(n)

dec

:18.28%/5%

250

(-100,0.01)

250

(-100,0.01)

?

(-500,0.01)

(-100, 0.2)

(-500,0.01)

●(-100,0.2)

200

·ration budget

●(-500,0.2)

ration tolerance

000

·ration budget

rationtolerance

●(-500, 0.2)

Treatment Group Size

Treatment Group Size

Treatment Group Size

150

150

150

100

100

100

(-100,0.01)

5

(-500,0.01)

(-100,0.2)

5

●(-500,0.2)

ration tolerance

ration budget

6

8

10

2

4

6

10

6

8

10

0008

(-400,0.01)

250

0008

·c=0.01

(-400, 0.2)

c=0.05

(-1500,0.01)

C=0.1

●(-1500,0.2)

200

●c=0.15

ration-budget Linkedin

c=0.17

Treatment Group Size

0009

Treatment Group Size

Treatment Group Size

0009

c=0.2

4000

4000

100

2000

·c=0.001

2000

c=0.007

c=0.01

c=0.02

c=0.05

C=0.1

8

10

00S

(-100,0.01)

0

(-100,0.01)

(-500,0.01)

(-100,0.2)

(-500,0.01)

（-100,0.2)

1000

400

(-500,0.2)

●(-500,0.2)

ration tolerance

009

ration tolerance

·ration budget

ration budget

00S

Budget Surplus

0

Budget Surplus

snjduns 1abpng

400

200

200

00S-

c=0.007

c=0.001

●c=0.01

●c=0.02

●c=0.05

000

●c=0.1

6

8

10

6

8

10

8

10

Budget-spent distribution To explore how our algorithms controls the risk of ruin and budget spending, we simulate following experiments for 5,000 times and plot distribution of the budget spent ∑ r ∈ [ t ] ∑ i ∈T r ( Y i , t (1) -Y i , t (0)) in Figure 1 (j)-(n): (i) norm : ( Y i , t (1), Y i , t (0)) i , t are sampled iid from (3) with µ true (0) = 0, µ true (1) = 1, σ (0) 2 = σ (1) 2 = 10 ; (ii) corr : same as norm except that for each i , t , Y i , t (1) is correlated with Y i , t (0) with correlation coefficient 0.8 (iii) bern : Y i , t (0) iid ∼ 6.4 Bern ( p = 0.5786) and Y i , t (1) iid ∼ 6.4 Bern ( p = 0.4224) ; (iv) fat : Y i , t (0) iid ∼ 1+ √ 5 t 4 , and Y i , t (1) iid ∼ √ 5 t 4 2 (v) dec: same as norm except µ true (1)( t ) = -( t -1) . Note that (iii), (iv) is configured so that E [ Y i , t (1) -Y i , t (0)] = 1, V ( Y i , t (0)) = V ( Y i , t (1)) = 10 . For all the above experiments, we run T = 10 stages with N t = 500, ∀ t and we use non-informative prior µ 0 ( w ) = 0, σ 0 ( w ) 2 = 100, w = 0, 1 .

As shown in Figure 1 (j)-(m) the model successfully controls risk of ruin for (i)-(iv). The actual ruin risk is at a reasonable level ( ∼ 1.2 %) compared to the ruin tolerance given ( 5 %). Note that the actual ruin risk are close for different outcome distribution. This is a consequence of central limit theorem and law of large numbers as discussed in Section 3.4. The model fails to control risk of ruin for (v) as expected since the treatment effect keeps decreasing and the model assigns treatment based on past stages which leads to higher-than-expected costs (cf. Section 3.4).

References

Apple. App store: Release a version update in phases, 2023.

Maryam Aziz, Emilie Kaufmann, and Marie-Karelle Riviere. On multi-armed bandit designs for dose-finding clinical trials. The Journal of Machine Learning Research , 22(1):686-723, 2021.

Eytan Bakshy, Dean Eckles, and Michael S Bernstein. Designing and deploying online field experiments. In Proceedings of the 23rd international conference on World wide web , pages 283-292, 2014.

Donald A Berry. Modified two-armed bandit strategies for certain clinical trials. Journal of the American Statistical Association , 73(362):339-345, 1978.

Iavor Bojinov and Somit Gupta. Online experimentation: Benefits, operational and methodological challenges, and scaling guide. Harvard Data Science Review , 4(3), 2022.

Iavor Bojinov and Karim R. Lakhani. Experimentation at yelp. Harvard Business School Case 621-064 , 2020.

Iavor Bojinov and Neil Shephard. Time series experiments and causal estimands: Exact randomization tests and trading. Journal of the American Statistical Association , page Forthcoming, 2019.

Asaf Cassel, Shie Mannor, and Assaf Zeevi. A general approach to multi-armed bandits under risk criteria. In Conference on learning theory , pages 1295-1306. PMLR, 2018.

Semih Cayci, Atilla Eryilmaz, and Rayadurgam Srikant. Budget-constrained bandits over general cost and reward distributions. In International Conference on Artificial Intelligence and Statistics , pages 4388-4398. PMLR, 2020.

Joel QL Chang and Vincent YF Tan. A unifying theory of thompson sampling for continuous risk-averse bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6159-6166, 2022.

David. R. Cox. Planning of Experiments . John Wiley & Sons, New York, 1958.

Debojit Das, Shweta Jain, and Sujit Gujar. Budgeted combinatorial multi-armed bandits. arXiv preprint arXiv:2202.03704 , 2022.

WFEberlein. On euler's infinite product for the sine. Journal of Mathematical Analysis and Applications , 58(1):147-151, 1977.

Nicolas Galichet, Michele Sebag, and Olivier Teytaud. Exploration vs exploitation vs safety: Risk-aware multi-armed bandits. In Asian Conference on Machine Learning , pages 245-260. PMLR, 2013.

Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesian data analysis . Chapman and Hall/CRC, 1995.

Google. Play console help: Release app updates with staged rollouts, 2023.

Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation . Pearson Education, 2010.

Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1168-1176, 2013.

Odalric-Ambrym Maillard. Robust risk-averse stochastic multi-armed bandits. In Algorithmic Learning Theory: 24th International Conference, ALT 2013, Singapore, October 6-9, 2013. Proceedings 24 , pages 218-233. Springer, 2013.

Jialiang Mao and Iavor Bojinov. Quantifying the value of iterative experimentation. arXiv preprint arXiv:2111.02334 , 2021.

Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. Batched bandit problems. The Annals of Statistics , pages 660-681, 2016.

Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology , 66(5):688, 1974.

Amir Sani, Alessandro Lazaric, and Rémi Munos. Risk-aversion in multi-armed bandits. Advances in neural information processing systems , 25, 2012.

Sama Shrestha and Sonia Jain. A bayesian-bandit adaptive design for n-of-1 clinical trials. Statistics in Medicine , 40(7):1825-1844, 2021.

Jerzy Splawa-Neyman, Dorota M Dabrowska, and TP Speed. On the application of probability theory to agricultural experiments. essay on principles. section 9. Statistical Science , pages 465-472, 1990.

Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 17-26, 2010.

Peter F Thall and J Kyle Wathen. Practical bayesian adaptive randomisation in clinical trials. European Journal of Cancer , 43(5):859-866, 2007.

Sattar Vakili and Qing Zhao. Risk-averse multi-armed bandit problems under mean-variance measure. IEEE Journal of Selected Topics in Signal Processing , 10(6):1093-1111, 2016.

Ryo Watanabe, Junpei Komiyama, Atsuyoshi Nakamura, and Mineichi Kudo. Kl-ucb-based policy for budgeted multi-armed bandits with stochastic action costs. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences , 100(11):2470-2486, 2017.

Tong Xia, Sumit Bhardwaj, Pavel Dmitriev, and Aleksander Fabijan. Safe velocity: a practical guide to software deployment at scale using controlled rollout. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSESEIP) , pages 11-20. IEEE, 2019.

Yingce Xia, Wenkui Ding, Xu-Dong Zhang, Nenghai Yu, and Tao Qin. Budgeted bandit problems with continuous random costs. In Asian conference on machine learning , pages 317-332. PMLR, 2016.

Yingce Xia, Haifang Li, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Thompson sampling for budgeted multi-armed bandits. arXiv preprint arXiv:1505.00146 , 2015.

Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. From infrastructure to culture: A/b testing challenges in large scale social networks. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 2227-2236, 2015.

Ya Xu, Weitao Duan, and Shaochen Huang. Sqr: balancing speed, quality and risk in online experiments. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 895-904, 2018.

Qiuyu Zhu and Vincent Tan. Thompson sampling algorithms for mean-variance bandits. In International Conference on Machine Learning , pages 11599-11608. PMLR, 2020.

A Decompose risk-of-ruin to individual stages

Proof of Theorem 3.1. The last claim is trivial: it is easy to verify that (1) and (2) hold if we let T t = ∅ for all t ∈ [ T ] . Now we prove the first and the second claim. The case for T = 1 is trivial. We assume T ≥ 2 . For any t = 2, . . . , T , we have that

where ( b ) used that P ( R t -1 > B | F t -1 ) = 0 ⇒ T t = ∅ and that r t (( Y i , t ) i ∈N t , ∅ ) = 0 , which implies that almost surely

and ( c ) used that b t ≥ B , which implies that almost surely

Rearranging this, we obtain a recurrence relation: for any t = 2, ..., T ,

Using the recurrence relation repeatedly for all t ∈ [ T ] , we obtain

as required. To prove the second claim, observe that equality is attained in all of the above inequalities if equality is attained in (14), ( i ), ( ii ) and ( iii ) , and that equality is attained in (14) if equality is attained in ( a ) and ( c ) . Finally, note that equality in ( a ) is attained if r t ≤ 0, ∀ t ∈ [ T ] and equality in ( c ) is attained if equality is attained in ( i ) and ( iv ) .

B Stochastic domination

LemmaB.1 (Stochastic domination under truncation) . For any two independent real random variable X , Z and real number a , t ∈ R such that P ( X < a ) > 0 , we have that

Proof of Lemma B.1 . Assume that P ( X ≥ a ) > 0 , or else the proof is trivial. We first claim that P ( X + Z ≥ t | X < a ) ≤ P ( X + Z ≥ t | X ≥ a ) . Note that this holds if and only if

The above holds since its lhs and rhs satisfies

It then follows from law of total probability that

as required.

Proof of Lemma 3.2. If M (1) t -1 = 0 , (8) holds with equality since S T t -1 (0) < S T t -1 (1) -B ⇐⇒ B < 0 . So, assume M (1) t -1 > 0 from now on. By (15f) and the conditional distributions of multivariate Gaussian, we have

glyph[negationslash]

where Z ∼ N (0, 1) is independent of S T t -1 (0) conditioned on F t -1 and µ , V are defined in (15f). Here, we used that V 11 > 0 since σ p , t (0) 2 , σ (0) 2 > 0 by Definition 3.1, and M (1) t -1 = 0 . Using the above and that S T t (0) = s T t (0) + S T t -1 (0) , we have

Since V 21 V -1 11 +1 > 0 in the above, using also that b t -S T t -1 (1), S T t -1 (1) -B ∈ F t -1 and that s T t (1) is independent of S T t -1 (0), S T t (0) , (8) follows from Lemma B.1.

C Derivation of the decision rule

Proof of these facts follows from standard Bayesian analysis (see e.g. [15])

Lemma C.1 (Posterior distributions) . We have for w = 0, 1, t ∈ [ T ]

where

D Robustness to non-identically distributed and non-Gaussian outcomes

glyph[negationslash]

Proof of Theorem 3.3. To show the experiment by Algorithm 1 is ( δ , B ) -RRC under Definition 3.4, it suffices to show that (1), (2) hold for each t ≥ 1 . Since (1), (2) hold for each t ≥ 1 if m t = 0 , we only need to show that for each t ≥ 1 , if m t = 0 , almost surely

glyph[negationslash]

Note that for each t ≥ 1 , if m t = 0 ,

where we used first inequality in (13) in (a), second inequality in (13) in (b), and (11) in (c).

glyph[negationslash]

We now show (16a) by induction. For t = 1 , if m 1 = 0 , Algorithm 1 ensures that

by construction, which implies that

glyph[negationslash]

almost surely. If m 1 = 0 , then P ( S T 1 (1) -S T 1 (0) > B | F 1 ) = 1 since B < 0 . This proves the base case. For the inductive case, if m t = 0 , Algorithm 1 ensures that

E

(

P

(

S

T

T

t

(1)

-

S

t

(0)

b

t

| F

by construction, which implies that

almost surely. If m t = 0 , we have that

from inductive hypothesis. This shows (16a).

To show (16b), note that under Definition 3.4,

On the rhs, s T t (1) -s T t (0) is independent of

and that S T t -1 (1) -B ∈ F t -1 . It follows from these, (16a) and Lemma B.1 that

glyph[negationslash]

Therefore, for each t ≥ 1 , if m t = 0 ,

as required. This concludes the proof.

t

)

| F

-

t

1

)

=

P

(

S

T

t

(1)

-

S

T

t

(0)

b

t

| F

-

t

1

)

∆

t

<

1

glyph[negationslash]

When are (13) satisfied Fix any t ≥ 1 where m t = 0 . Note that

The summands on the rhs are independent random variables under Definition 3.4. We thus expect that when m t or M t -1 are sufficiently large,

by central limit theorem under mild moment-growth conditions (e.g. Lyapunov's conditions). We thus expect that first condition in (13) holds when m t or M (1) t -1 are sufficiently large for each t ≥ 1 . We now focus on the second condition in (13). Suppose that ∆ t ≤ 0.5 , which implies z t ≤ 0

by (11). Note that we can write

and

For t = 1 , and

So, second condition in (13) holds for t = 1 if we have chosen prior and model parameters such that

This corresponds to that we choose prior and model parameters conservatively in the sense that we do not overestimate treatment effect or underestimate its variability. Now fix any t ≥ 2 . From the law of large number, we expect that for M (1) t -1 sufficiently large

So if the treatment effects increase or stay roughly constant throughout the experiments

and our variance estimates σ (0) 2 , σ (1) 2 are accurate or conservative in the sense that

the second condition in (13) holds for each t ≥ 2 and the experiment produced by Algorithm 1 is ( δ , B ) -RRC.

E Algorithm for general Bayesian models and costs

The following outcome model is a generalization of Definition 3.1. Here, experiment outcomes are allowed to be multivariate with each coordinate corresponds a different business metric.

Definition E.1 (General Bayesian model) . Fix p , q ≥ 1 . The model parameter θ true ∈ R q is generated from certain prior π 0 . The experiment outcome of unit i at stage t are distributed independently and identically as

where Y i , t (0), Y i , t (0) ∈ R q and p ( θ true ) is a probability distribution on R p × p .

The following is a generalization of Definition 2.1. It allows for general experiment cost beyond treatment effect. The cost of treating unit i is now h it = h t ( Y i , t (1), Y i , t (0)) for some function h t : R p × p ↦→ R chosen by the user. For instance, h t can be chosen to compute the worst treatment effect across multiple business metrics.

Definition E.2 (General experiment cost) . For each t ≥ 1 , let the experiment cost from staget and treated unit i be h it = h t ( Y i , t (1), Y i , t (0)) where h t : R p × p ↦→ R is any user-chosen function. Then define r t := ∑ i ∈T t h i , t . We let r t = 0 if T t = ∅ . Define the cumulative experiment cost up to stage t as R t := ∑ k ∈ [ t ] r k .

We now move to derive an explicit algorithm Algorithm 1 from Theorem 3.1 that output ( m t ) t ≥ 1 such that the experiment is ( δ , B ) -RRC. Compared to Algorithm 1, the algorithm developed in this section will require Monte-Carlo simulations and generally gives more conservative ramp schedule.

We first review the Cantelli's inequality, which is an improved version of the well-known Chebyshev's inequality for one-sided tail bounds.

Lemma E.3 (Cantelli's inequality) . For any λ ≥ 0 , and real-valued random variable X with finite variance,

Given that (i) P ( R t -1 ≥ B | F t -1 ) > 0 and that (ii) E [ R t | R t -1 ≥ B , F t -1 ] ≥ b t , a direct application of Cantelli's inequality shows that

where F 0 denotes trivial σ -algebra.

Our strategy to construct an algorithm that selects ramp size m t such that (1), (2) hold is as follows: we first verify that condition (i) holds; if not, set m t = 0 and otherwise find m t such that the following two inequalities hold

To accomplish this, note that by exchangeability of the outcomes under Definition E.1,

and

We thus require a Monte-Carlo procedure to output estimates ˆ ϕ t (0), . . . , ˆ ϕ (6) t for the following posterior quantities on the rhs of (18), (19)

where h i =1, t , h i =2, t denote costs from treating two units i = 1, 2 at stage t . Recall that under (...), the outcome of the units are exchangeable. So i = 1, 2 simply refers to any two distinct units. These quantities will be used to construct estimates of E [ R t | R t -1 > B , F t -1 ] and V ( R t | R t -1 > B , F t -1 ) as functions of m t chosen.

We now outline a procedure to construct ˆ ϕ t (0), . . . , ˆ ϕ (6) t . Firstly, suppose that we can obtain K samples from the posterior distribution

from certain MCMC algorithms. The specific details of the MCMC algorithm will depend on the Bayesian model used, but generating posterior-predictive samples while imputing unobserved data, as required in (20), is a common objective of such algorithms (see e.g. [15, Chapter 18]). Let us denote the K samples as

These will give us K samples from [ h i =1, t , h i =2, t , R t -1 | F t -1 ] as follows:

Then we can estimate P ( R t -1 ≥ B | F t -1 ) by

Let

which denotes the subset of the K Monte-Calor samples for which the budgets are not depleted.

If ˆ ϕ t (0) = 0 ⇐⇒ L t = ∅ , we can simply out m t = 0 since this corresponds to the case that the condition (i) does not hold, i.e. P ( R t ≤ b t | R t -1 > B , F t -1 ) ≈ 0 . Otherwise, we continue to construct ˆ ϕ t (1), . . . , ˆ ϕ (6) t as follows:

From (18), (19) and the Monte-Carlo estimates above, we then have estimators for E [ R t | R t -1 ≥ B , F t -1 ] , V [ R t | R t -1 ≥ B , F t -1 ] in terms of ˆ ϕ t (1), . . . , ˆ ϕ (6) t as follows

The two inequalities in (17) then become

respectively. Assume that ∆ t > 0 or else set m t = 0 directly. Observe that (23b) can be written as, with q t := ∆ -1 t -1 , where

Then one can choose m t to be the largest, positive integer in the range defined by

If the range does not contain any positive integer, we set m t = 0 . Note that the range can be easily identified after solving the quadratic equation A t m 2 t + B t m t + C t = 0 . Algorithm 2 gives the algorithm that outputs ramp sizes adaptively. Note that by construction, it gives a ( δ , B ) -RRC experiments if the Monte-Carlo estimators are sufficiently accurate.

Algorithm 2 Output ramp size adaptively

glyph[negationslash]

We have conducted preliminary simulations of the proposed procedure for a multivariate Gaussian outcome model with Gaussian-inverse-Wishart prior, and observed satisfactory results. However, we defer presenting numerical results until future work when a more systematic investigation of Monte-Carlo based procedures can be conducted.

F Linkedin experiment data

In Table 1 below, µ true ( w ), σ ( w ) 2 , w = 0, 1 are sample statistics from the actual LinkedIn experiment. N t are incoming population size reduced by 10 4 factor for tractability on a personal computer.

G Thompson-sampling based Bayesian bandit

This algorithm is developed in [27, Section 4] for clinical trials. The algorithm assigns a user i at stage t ≥ 1 to treatment with probability

for tuning parameter c > 0 . Under Definition 3.1, by (15d), we have that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with a problem of gradually releasing a resource to a population modeled as a risk-of-ruin constrained experiment. Namely, at every stage $t$ we have an arriving population $\\mathcal{N}_t$, and we need it into a control and treatment groups $\\mathcal{C}_t$ and $\\mathcal{T}_t$ respectively (the treatment group is assumed to be the one receiving the resource). However, this resource allocation can potentially come at some cost, which depends on the population receiving the treatment. The goal of the problem at hand is to keep the overall such cost under a specified threshold (budget constraint), while minimizing the number of rounds/stages it takes to cover at least half of the underlying population.\n\nAs the authors nicely explain, this model has natural applications in the phase release of software products/updates.\n\nFor this problem, the authors give an adaptive Bayesian algorithm that satisfies the budget constraint with probability $1-\\delta$. Namely, given any confidence $\\delta$, the algorithm achieves the desired guarantees with probability $\\geq 1-\\delta$.\n\nSTRENGTHS:\n1) The problem seems well-motivated.\n\n2) The paper presents both sound theoretical results and a validating experimental evaluation.\n\n3) To my understanding (and take this with a grain of salt since I am not knowledgeable in this area) based on what the authors mention in their related work, devising a structured algorithmic way of dealing with the phase release problem in this way is novel.\n\nWEAKNESSES:\n1) I found the presentation a bit incomplete and difficult to follow at certain points:\n\na) Are the arriving subpopulations between different stages disjoint?\nb) When you say that the \"stopping condition\" is covering half of the population, does that mean half of $\\cup{t \\geq T}\\mathcal{N}_t$?\nc) Besides defining the RRC experiment, it would be useful to formally define the whole problem, including the optimization objective.\nd) It would be nice to give some further motivation behind the experiment cost in Definition 2.1. Why is this cost capturing something meaningful?\n\nQUESTIONS:\nLook at the previous section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

comments: SUMMARY:
This paper considers the problem of phased releases and formulates the problem into a batched bandit.

STRENGTHS:
This paper is very well-written and  the problem very motivated. It is great to see bandit algorithm to solve a real-world application rather than stay in the theory world. The algorithm is fairly simple and this paper is well-executed with theory and experiments. I am glad to see the semi-real linkedIN experiments.

WEAKNESSES:
The algorithm is fairly simple and the theory is routine. Well for an application paper, we should not expect the algorithm to be quite complex.

QUESTIONS:
no

LIMITATIONS:
no
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an algorithm for conducting automated phased release strategies that balance risk and reward by controlling the risk of ruin while maximizing ramp-up speed. The authors propose a framework that models the problem as a constrained batched bandit problem and uses an adaptive Bayesian approach. The algorithm is designed to be efficient and parallelizable, and is claimed to be robust to model misspecifications.\n\nSTRENGTHS:\nOriginality:\n\nThe tasks and methods presented in the paper are relatively new and provide a novel approach to phased release strategies in the technology industry.\nThe work combines well-known techniques, such as constrained batched bandit problems and adaptive Bayesian approaches, in a unique and valuable way.\n\n\nQuality:\n\nThe submission is technically sound, with the proposed algorithm built upon a solid theoretical foundation.\nThe claims are well supported by both theoretical analysis and experimental results, including simulations and a semi-real LinkedIn data experiment.\n\nWEAKNESSES:\nClarity in Section 3: The authors introduce the concept of the risk-of-ruin-constrained (RRC) experiment, but the problem definition and objectives of the proposed algorithm are not explicitly presented. In the abstract and introduction, the authors mention to balance risk control and maximize ramp-up speed. However, the paper lacks a clear explanation of how the algorithm's design specifically contributes to maximizing ramp-up speed. The algorithm's goal appears to be to output the treatment group size while satisfying the budget constraint. A clearer problem definition and a more direct explanation of the algorithm's goals would help readers better understand the problem being addressed and the significance of the proposed solution.\n\nMaximizing ramp-up speed: In the experimental results, the authors mention that a larger budget (B) and higher risk tolerance can lead to faster ramp-ups. However, the connection between the algorithm's design and its ability to achieve faster ramp-ups is not explicitly discussed. Upon a closer examination of the paper, the authors propose a decomposition scheme in Theorem 3.1, which breaks the risk constraint into stage-wise constraints. This approach helps control the current-stage cumulative experiment cost, given past observations, and determines the treatment assignment based on the posterior inference of the remaining budget. However, the paper could further elaborate on how these stage-wise constraints or the algorithm's design contribute to optimizing ramp-up speed.\n\nQUESTIONS:\nIn Figure 1 (b), the line plot does indeed reach close to zero and then increase. This behavior can be attributed to the algorithm adapting to the realized outcomes. When the outcome is negative, the algorithm may still use the remaining budget to explore further, rather than stopping the experiment immediately. It is possible that the algorithm is designed to learn from past observations and adjust treatment assignments accordingly, which could explain this behavior. However, it would be helpful if the authors provided more insights into the reasons behind this behavior and whether any modifications could be made to the algorithm to address this concern.\n\nIn line 172, the mention of \"some prior estimate\" is not specific about which parameter should be determined first. The authors could clarify this by providing examples of which parameters are typically estimated in practice, such as the treatment effect or the variance of outcomes.\n\nIn line 5 of Algorithm 1, it appears to be a typographical error. The correct values for w should be 0 and 1 (i.e., w = 0,1), as these values correspond to the treatment and control groups in the paper.\n\nLIMITATIONS:\nRevise Section 3 to provide a clearer problem definition and discuss the objectives of the proposed algorithm in more detail.\n\nExplain the design aspects of the algorithm that contribute to maximizing ramp-up speed, and how these aspects are balanced with the need to control risk. The paper could benefit from additional experimental results that compare the proposed algorithm with baseline methods in terms of ramp-up speed under the same violation rates. This would help demonstrate the algorithm's effectiveness in balancing risk and quickly ramping up compared to other methods. The authors may consider conducting more experiments or simulations to showcase the superiority of their approach in this regard.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors address the problem of finding a risk-sensitive strategy for phased releases. A model that involves a risk budget is proposed and an algorithm based on Bayesian updates is presented to find a solution. The proposed algorithm is empirically tested on a range of problem setups and is shown to outperform existing bandit algorithms.

STRENGTHS:
The problem setup and the proposed model for phased release seem novel to me. Although this is arguably a rather narrow and specific problem, the authors do contribute some useful ideas in applying the Bayesian approach to a challenging problem, which could be valuable to the NeurIPS community. The paper is easy to read and overall presentation is clear.

WEAKNESSES:
The proposed model is simple enough to avoid the need for tedious sampling-based solutions. However, one wonders whether some of the parameters (such as the risk budget) can be easily instantiated in real-world scenarios. The i.i.d. assumption simplifies the analysis but again one wonders whether it would lead to overly conservative or overly risky solutions in practice.

QUESTIONS:
Page 9 L259 I suppose this is NPTE rather than PNTE. It does make me wonder why there is no PNTE scenarios in the experiments.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07