When to Make and Break Commitments?
Authors: Alihan Hüyük
Zhaozhi Qian
Mihaela van der Schaar
Venue: ICLR 2023 poster (2023)
Keywords: optimal stopping/switching
sequential hypothesis testing
adaptive experimentation

Abstract: In many scenarios, decision-makers must commit to long-term actions until their resolution before receiving the payoff of said actions, and usually, staying committed to such actions incurs continual costs. For instance, in healthcare, a newly-discovered treatment cannot be marketed to patients until a clinical trial is conducted, which both requires time and is also costly. Of course in such scenarios, not all commitments eventually pay off. For instance, a clinical trial might end up failing to show efficacy. Given the time pressure created by the continual cost of keeping a commitment, we aim to answer: When should a decision-maker break a commitment that is likely to fail—either to make an alternative commitment or to make no further commitments at all? First, we formulate this question as a new type of optimal stopping/switching problem called the optimal commitment problem (OCP). Then, we theoretically analyze OCP, and based on the insights we gain, propose a practical algorithm for solving it. Finally, we empirically evaluate the performance of our algorithm in running clinical trials with subpopulation selection.

Published as a conference paper at ICLR 2023
WHEN TO MAKE AND BREAK COMMITMENTS?
Alihan Hüyük
Zhaozhi Qian University of Cambridge 2224 4 Caam. ac.uk uk
Mihaela van der Schaar University of Cambridge The Alan Turing Institute mv 72 cam. ac uk
University of Cambridge ah20750 cameccamaccmm uk
ABSTRACT
In many scenarios, decision-makers must commit long-term actions until their resolution before receiving the payoff of said actions, and usually, staying com- mitted such actions incurs continual costs. For instance, in healthcare, a newly- discovered treatment cannot be marketed to patients until a clinical trial is con- ducted, which both requires time and is also costly. Of course in such scenarios, not all commitments eventually pay off. For instance, clinical trial might end up failing to show efficacy. Given the time pressure created by the continual cost keeping commitment, we aim to answer: When should commitment that is likely to fail-eithe to make an alternative commitment or to make no further commitments at all? First, we formulate this question new type of optimal stopping/switchie (OCP). Then, we theoretically analyze OCP, and based on the insight we gain, propose a practical algorithm for solving it. Finally, we empirically evaluate the performance of our algorithm in running clinical trials with subpopulation selection.
break a
problem called the optimal commitment problem
INTRODUCTION
In many real-world settings, decision-makee must commit long-term actions and wait until their resolution before receiving the payoff of said actions. Meanwhile, staying committed to such actions incurs continual costs. For instance, in portfolio management, it might take time for an asset to develop additional value after an initial investment, and keeping capital tied up in an asset comes with an opportunity cost for the investor (Markowitz 1959 Merton 1969 Karatzas and Wang 2020). In an energy network, turning power stations on and off is not an immediate action, hence a sudden increase in energy demand can only be met with a delay after putting more stations into operation. and keeping stations operational obviously consumes resources (Rafique and Jianhua 2018 Olofsson al. 2022). healthcare, a newly newly-discovered treatment can only be marketed to patients once a successful clinical trial that targets the said treatment is conducted, which both requires time and is also costly (Kitiin 2010: Umscheid et al. 2011).
Of course, not all commitments eventually pay off: An asset might end up losing value despite investments, energy demands might shift faster than a network can react to, and a clinical trial might fail to show efficacy for the targeted treatment. Given the time pressure created by the continual cost of keeping a commitment, our goal in this paper is to answer the question: When should a decision-maker break a commitment-therehh avoiding future costs but also forfeiting any potential returns-either to make an alternative commitment instead or to make no further commitments at all? Solving this problem optimally requires a careful balance between exploration and exploitation: The earlier a commitment that is bound to fail is broken, the more resources would be saved (cf. exploitation): but the longer one kept, the more information is revealed regarding whether the commitment is actually failing or might still succeed (cf. exploration) and in certain cases, also regarding the prospects of similar commitments one could make instead.
Related problems are mostly studied within the context of adaptive experimentation and sequential hypothesis testing (see Section 5). As such, we focus on adaptive experimentation as our main application as well. More specifically, we consider the problem of selecting the target population of an adaptive experiment. Suppose an experimenter, who is interested in proving the efficacy of a new treatment, starts running an initial experiment that targets certain population of patients. Incidentally, the treatment being tested is effective only for relatively narrow subpopulation of patients but not for the wider population as whole. Hence, experiment targeting the overall population, but not the subpopulation specifically, will most probably fail to prove efficacy and prevent the deployment of the treatment for the patients who would have actually benefited from not to mention waste

---

Published as a conference paper at ICLR 2023
time and resources (Moineddin et al. 2008 Lipkovich al. 2017: Chiu al. 2018). Of course, the experimenter has no knowledge of this in advance but the initial experiment they have set up would slowly reveal more information regarding the effects of the treatment and the fact that the ongoing experiment is bound to fail. In that case, we want to be able to determine at what point the experimenter has enough information to justify breaking their commitment to the initial experiment that targets too wide of a population to be successful, in favor of making new commitment to a follow-up experiment that focuses on a narrower subpopulation instead?
Contributions Our contributions are threefold: First, we formulate the problem of making and breaking commitments in a timely manner as a new type of optimal stopping/switching problem called the optimal commitment problem (OCP) (Section(2) The defining feature of OCP is that rewards are received only when a known time point reached but costs are incurred continually, requiring commitment to actions but with incentive to abandon those commitments. As we will show later, OCP cannot be easily solved via conventional reinforcement learning techniques due to its non-convex nature. Second, we theoretically analyze a simplified case of OCP to identify the characteristics of the optimal solution (Section 3 and based on the insights we gain, propose a practical algorithm for the more general case (Section[4) 4). Third, we empirically evaluate the performance of our algorithm in running experiments with subpopulation selection (Section (6). Before we move on, it should be emphasized that, although we predominantly consider adaptive experimentation as our main application, our contributions remain generally applicable to portfolio management, energy systems, and any other decision- docision-makkin scenarios that require commitments to long-term actions.
2 OPTIMAL COMMITMENT PROBLEM
We first introduce the problem of optimal commitment from the perspective of running experiments. As far formulation is concerned, experiments are conducted to confirm the efficacy of an inter- vention by observing the outcome of the said intervention for subjects belonging to a particular popula- tion. However, this experiment-focused perspective does not limit the applicability of OCP; we stress its generality later at the end of the section. We provide a glossary of terms and notation in Appendix [K]
Let X' be a discrete set of atomic-populations such that every subject is only the member of exactly one stomic-populatie subject being from coomic-population x (such that E 1), and with U. the distribution of outcomes for atomic- pooniiipoplltiinn...m x such that the mean outcome 0₂ = Ey--e, [y] is the effect of some intervention for atomic- -oopiiitooolttiionn x. Now, wider populations can be constructed by combining various atomic- coaniccpopulations. Let any X C X' represent the population of subjects who belong to either {ar E X). Then, the probability of a subject being from population X can be written as 1ix Emex Nw. the probability of a subject being from atomic-population conditioned on the fact that they are from population X can be written as 7// = 7//Txx, and the average effect for population X can be written 0x ExexNNu|X0
Populations
x E X. Denote with E [0, 1] the probability of a
one of the atomie-populatinn
Experiments An experiment is largely characterized by the population it targets, its sample horizon, and its success criterion. During experiment that targets population X, each time step t E .I. that the experiment continues, first a subject from some atomic- Moomic-population Tt within the targeted population X arrives with probability 7//i,, and then the outcome y/ Ωr, for that subject is observed. This process generates an online dataset D; = {ze . J The experiment terminates when pre-specified sample/time horizon 7 is reached. Once terminated, the experiment is declared a success if ) = 1, where p (X' x IR) 7 - {0, 1} is the success criterion, and declared a failure otherwise. Formally, the tuple vi (X, T, p) constitutes an experiment design.
eettrexpprrmenter Suppose a meta-experimmnter is given a set of viable experiment designs V and is tasked with running least one successful experiment. Each experiment va E has an associated cost C₁ E R4. which the experiment incurs per time step that it continues, and an associated reward Rgg E RR,, which the experiment provides only if it eventually succeeds. The meta- experimenter aims to maximize utility-that is the difference between any eventual reward received and the total costs incurred by running experiments. They first pick an initial experiment ei² E v and start conducting it, which generates an online dataset DJ as described earlier. Now at each time step f, they need to decide whether they should stay committed to their initial decision and wait until Tb' terminates, or stop v early in favor of starting new experiment 3 They might decide on the latter to avoid unnecessary costs if D. already indicates & is unlikely to succeed. If at some point secondary experiment 2 is started, now the meta-experiment has similar decision make
2

---

Published as a conference paper at ICLR 2023
regarding whether to stop 2 early in favor of starting new experiment 3 V. This process continues until either an experiment finally succeeds or the exetrrexpei decides not to conduct any further experiments; let the random variable
(1,2.... ...} be such that 16 is the last experiment. We denote with v/* = =xx; r', p²) the i-th experiment conducted by the meta-experimmnter and with T the number of time steps for which the i-th experiment is conducted either until it was stopped by the meta-experimenter or the time horizon r' was reached. Denote with T(t, v Di) the decision-making policy of the meta-experimenter, where t is the current time step of the latest experiment v² and Di = (U:=DD), U Di is an aggregate dataset. We write (i) (t. vh DD) = 1e if the meta-experiment decides to keep conducting the current experiment 1. (ii) (t. 16 D)) = v j n/² if the mnpa-experimenter decides to stop experiment v/² and start experiment vy instead, and (iii) nT(, w/,, Dj) = Ø if the meta-experimmnter decides not to conduct any further experiments.
Objective Once all experimentation is concluded, the meta-experimenter achieves the total utility == Ra 11T" = = -D" Ei= -
(I)
Then, the optimal commitment problem is to find the optimal policy π²' = argmax, E, [G] that maximizes the expected utility given V, {Mzz). (Ra, C without knowing mean outcomes {0,) or outcome distributions {Sz]] It is called the optimal commitment problem because each exper- iment & (X, T, p) only provides a reward if the meta-experimenter commits incurring its costs for at least T time steps, and the meta-experimenter needs to decide which experiment in y the better commitment or if there is any experiment worth committing to at all adaptively.
General applicability of OCP Although we have de- scribed OCP from the perspective of sexeta-)expermmenn design, it can potentially be useful in modeling many other problems as we have stressed during the introduction (see Table D. For instance, in portfolio management, atomic- populations can be regarded as various assets one can in- vest in, then population would correspond to portfolio assets. Similar to experiments, when these portfolios require a time commitment (cf. 7) before they provide their payoff (cf. R&)) and incur an opportunity cost (cf. C..) in the mean time, the decision- making problem of managing when and which portfolio to invest in constitutes an instance of the opti- mal commitment problem. Another good examples is energy management, where power stationy and the networks they form are akin to atomic poppla-populations and populations. Since power stations cannot be turned on and off immediately, putting one in operation requires certain amount of commitment.
Table 1: Equivalent concepts across dif- ferent domains. OCP can model scenarios other than adaptive experimentation.
Damain
Equivaleat Coucepis
Adaptive expecimentatiin
Atomic population
Population
Poetioll monagement
Financial asset
Pontolio
Ensay sysleme
WARM-UP: WHEN TO BREAK A SINGLE COMMITMENT?
In this section, to gather insights, commence by analyzing simplified instance of OCP. Later, in Section [4] using these insights, we construct a practical algorithm for solving a more general case of OCP. As the simplified instance, only consider atomic-population such that X' = (Nn and one experiment design that targets this atomic-populatee Moreover, we assume that the outcomes are distributed normally with unit variance such that Ω = Ωx. = NO 0x 1) and the success criterion is a simple Z-test to see whether 0 >>> 0 such that p(D,) = p(u+)) = 1 (ur a/vT), where μ4 Eiey, E.,) Ve / /D,] is the empirical mean outcome given dataset D., and a determines the significance threshold for the test. Since there is just one viable experiment in this setting, the only decision that needs to be made at each time step is whether to keep conducting experiment v Vo or to stop all experimentation. For this decision to be interesting, we will also assume that C = Co. > 0-so so that never stopping is not necessarily optimal- and R = Rv,, > -C-so that always stopping is not necessarily optimal either.
such that v = (v, = (Xo, T, p)).
Value and Q-functions Since and μt are sufficient statistics to estimate the success probability of the experiment, it is also sufficient to only consider policies of the form x(t, n). For a given policy n,
V (t, ()) = E[R. 1{T2 > p(p-) C (min/T7 .T] -
[4]
(2)
= [ 1(T21 > T) -0(x)))))= - C (min([211.7 p4 1x
are the value function, and the Q-function of conducting the experiment for at least one more time step respectively, where T7 = min(t' > t: s(t', )r) = 2) is the first time step at or after time t that policy æ decides to stop; let V = V= and Q* Q" be the optimal value and Q-funciions. Note that the fafcttor of stopping all experimentation is always equal to zero for all policies. Hence, the optimal policy must be such that π² rt, ()) Vo Q" (t., 1)) > and nT n'(t, p1) = 2 otherwise.
3

---

Published as a conference paper at ICLR 2023
Once we identify the value and Q-functions, naive at- tempt at finding the optimal policy would be to compute V and Q" via dynamic programming they would sat- isfy the following Bellman optimality conditions:
Optimal Wan (V')
P7 2.
P7 3,40
(t, p)) = C' + (t + 1, (++1)) = μl (" = max (t,
(4)
(5)
115
-1
0355
0.5
1.5
and (T) R p(n). However, a major complication in applying dynamic programming methods to compute V* and Q* is that they are continuous functions in μr. In the literature of partially-observable Markov decision pro- cesses (POMDPs), which OCP happens to be an instance of (see Appendix A. the standard approach of addressing this complication would be to leverage the convexity of V and Q", and approximate them with functions of the form f(u) max; a:ji + b; (Spaan 2012). However, this standard approach is not applicable in OCP because, in general, neither (*(,, ()) nor (t, p.) is convex function with respect to j (see Figure D::
Test Statistic (o)
Figure 1: Optimal value function V (t. p4)
for C = 1, R 10, T 4, and a = 0. It can clearly be seen that neither V V is convex in μ (cf. Proposition 1).
Proposition 1 (Non-convexity). There exist a problem instance (C, R, T, a)) and L E such that 3p. E P.p E 0, 1] V* (t. pp + (1 -11-pp0 p)y")
(t, +11 pl (t, and Sp, p' E R.p p E [0, - (t, pp + (1 ppppppp)))))) < <pp (t. (1 (1- (t,"
Properties of the optimal policy Although identifying xt exactly by computing V and Q" is challenging, we can still identify some properties that x² should have, which can then help us design a heuristic policy that we expect to perform well, albeit not optimally. First of all, the optimal policy x² should be ""hresholding-yype" policy-that is the meta-experimenter should keep conducting the experiment as long as br stays above a time-dependent threshold μa and should stop all experimentation the moment μ4 drops below that threshold (see the top panel of Figure (
Proposition 2 (Thresholding). For all problem instances (C, R,T,a), there exists time-dependent thresholds {uī E [=] such that
if Mi; x' (t. (V
Ø otherwise)
Intuitively, a higher test statistic l4 means that the exper- iment only more likely to succeed, hence if it is optimal
degion
Teu Stativic (x)
to continue conducting the experiment when f4 = 14, then it should also be optimal to continue when 14: = y > μ (likewise, lower d4 means success is even less likely hence T² (t, p)) = Ø implies x " n²) = Ø for pl < 1)).
0.5
regioss buth and cop
1.0
Throshold p Thushold pf
-1.5%
Moreover, the optimal policy x" must be "optimistic" that the experiment will succeed when making decisions. Con- sider greedy policy grreddy that continues as long as the expected utility of committing fully to conducting the ex- periment until it terminates f groody = Vo if and only if V "0) (t. μu) > 0 where (0) is the policy that always waits until the experiment termi- nates such that - (t, p)) = Vn for all t, H;, greedd is said to be greedy because the decision to continue is made as- suming full commitent to the experiment without con- sidering the possibility stop at future time step. Then, whenever such greedy reasoning suggests continuing. the meta-experimenter should indeed continue. However, whenever the same reasoning suggests stopping, the meta- experimenter should be optimistic that the experiment will succeed and occasionally make the decision to continue instead that is ;²" should be biased towards continuing (see the threshold gap in Figure 2
1.6
True dureshidd gap by Upper bound in Propostion
1.0%
0.5
is positive-thar is
Timo Step (1)
Figure 2: Visualization of Propesitions 2-4
for C = R 50, T 18, and Top figure shows how the thresholds in Proposition Deesittionnevoovee evoolve over time. Notice that ei S 14. L.grodly hence why "' optimistic (cf. Proposition D. Bottom figure shows how the optimism of measured as lut u decreases over time (cf. Proposition
uredly
Proposition (Optimism). First, aomedy is also of thresholding type and there exists {nr greedy E &R such that grreed (1, 14) Wo if and only if > 1 greedy Moreover, for all t E (1,...,, 13.
1 S 1h
greedy
{p: : n (1, vo} 2 {u: : n Breedy (t, = =V03
Proofs of all propositions are given in Appendix i
4

---

Published as a conference paper at ICLR 2023
Intuitively, the optimism of '-" accounts for the information gained from observing more samples when the experiment is continued. Remember that sreedy estimates the reward to be received if the experiment is conducted until termination, and it stops whenever its estimate is negative. But, the estimate of zrreddy has some uncertainty associated with it. Whenever it is uncertain enough that the reward to be received is actually negative; incurring the cost of continuing for one more time step, gaining new information, and forming more certain estimate can lead to a more accurate decision and a higher overall utility. Finally, the optimism of I has a strictly decreasing upper bound; denoting with F(x) = (1/v 2T) -=x0 c- (1/2)a² the c.d.f. of the standard normal distribution:
Proposition 4 (Decreasing optimism). For all t E TT....,T--1].
lui - - | S 1/r x (F- ---())))))(
Intuitively, as the experiment continues, the information gained from one individual sample decreases relative to the total information accumulated, hence the optimism of x" that accounts for the that information gain also decreases (see the bottom panel of Figure 2). Consider one extreme: When
t = T - 1, there is no more information to be gained before the experiment terminates at t = T, hence r" should make the same decisions as grredd. Indeed, Proposition 4 implies that 1. 1
4 A PRACTICAL ALGORITHM: BAYES-OCP
Summarizing discussion in the previous section, we suspect the optimal policy to be (i) of thresh- olding type (cf. Proposition 2). (ii) optimistic (cf. Proposition B) and (ii) increasingly more greedy (cf. Proposition 4 These findings
not a complete surprise as optimismm-intthe-faee fffbh---uccc----f-merttaaa is a well-known principle in solving online decision-making problems (Auer et al.] 2002 Bubeck et al. 2012). Our earlier analysis shows rigorously that this principle holds for at least special case of OCP and strengths our intuition that it should be applicable for more general cases of OCP as well.
Keeping properties (i-iii) in mind, we now propose a practical algorithm for solving OCP in more general setting than the one we analyzed earlier. Let |X| 2 1 and V/ = {(X, p) X E 2 \ ( include all experiment designs that target a unique subpopulation within X' for given time horizon and success criterion p; let Cx - C(X,7.p) and Rx = R(x, We assume that the conditional power of performing hypothesis test at time T according to top p-that is the probability of the test being successful conditioned on mean outcomes {0,] can be computed for interim datasets that is
P(X, Dúi {0+})= Exp-in x whex N,r~N(0,,...1 U (U/=441 txe, be }))
(9)
can be evaluated efficiently. Then, based on this conditional power function, we define
Di, - = P()
Algorithm 1 Bayes-OCP
(10)
I: Initialize μa and of for all æ E X
as the expected utility of fully committing to an experiment and waiting until it terminates when the experiment targets population X, is currently at time step t, and has collected dataset D. so far.
2: X X, t 0, Do + Ø
3: Start experiment 36 T.
4: loop:
141+1, D, < D. U (I. 1/0 + 1/00,, +1
6:
Denote with g(o)(x; (0.)) g(X, Φ0; (0z)) the same expected utility but for an experiment that is yet start, and with (()) (Ø; (0.))
7: [a, -aa + (yr 12
(i) Identify candidate subpopulation X' to replace X:
8:
X' +2
the utility of stopping all experimentation.
9:
while X X'' 30
Our algorithm is called Bayes-OCP and is given in Algorithm| T It maintains a posterior distri-
10:
x² argmax,exx
Eo (g (X' (x)))
bution (p 2 for each mean outcome oz assuming that, given mean 6z, outcomes are distributed normally with unit variance- that is Ω2 = (e,,,, 1). These posteriors are only used in deciding which experiment to run next and not in determining whether the experiment was a success or not. Hence, even when the assump- tion of outcomes being normally distributed is violated, the integrity of the experiments would not be effected; only the performance of Bayes-
11:
if B. BoDNNin of) ( (X' fx') (0.))
> Ee Bo g (0.)))
12:
X' + x' X''O (a
13:
else: break
(ūi Decide whether to actually replace X with X':
14:
if If 2) (0)0) (X': (0z)) > -(x, Dr: (0.))) > B:
15:
X'. t 0. Dn + 2
16:
Start a new experiment = (X,
5

---

Published as a conference paper at ICLR 2023
OCP in managing various experiments would degrade (see Appendix C for related experiments). Making use of the posteriors it maintains, Bayes-OCP performs two steps at each iteration:
(i) First, a subpopulation X' C. X within the currently targeted population X is identified as potential candidate to target next; due to the combinatorial size of V. it would not be prac- tical to consider every subpopulation individually as candidate for large |x(). The ideal candidate would be the subpopulation with the largest expected utility: X" = argmaxx cx Ee,~Nip, ot) (t) [g(0) (X'; [(0)))]] But again due the combinatorial size of the search space, Bayes-OCP employs greedy algorithm instead and forms candidate subpopulations by com- bining, one by one, the atomic-sin coomicsubbopullatinn
that increase the expected utility the most, until the expected utility no longer improves. Note that it is common to use greedy algorithms to solve combinatorial optimization problems (Lawler 1976 Papadimitriou and Steiglitz. 1982).
(ii) Then, it is decided whether the current experiment targeting population X should be stopped in favor of targeting candidate X' identified earlier instead. A greedy strategy would have done so whenever (H) (g) (X'; {0z > (o).)N((==.0))G((3
D;; [02 But from our earlier analysis, we have learned that the optimal strategy is optimistic (cf. Proposition B As such, Bayes-OCP checks whether it is overwhelningly likely that the alternative experiment has higher expected utility-that is whether o2) (g (X'; {0x)) > g(X, De; (0x))) > 8. where 3 E (1/2, 1) controls the decision- decision-making threshold. When 3 is large, we are more optimistic that the current experiment will succeed and require stronger evidence that the altema- tive experiment has higher expected utility. Note that, as the posteriors N(uz, o get narrower, the optimism of this rule naturally decreases, which should the case for the optimal strategy (cf. Proposition 4p. As one extreme, the two switching rules become equivalent when {o? + 0}.
RELATED WORK
Optimal stopping Optimal commitment is essentially new type of optimal stopping/switching problem. In typical optimal stopping problems (OSPs), the reward an agent can receive evolves based on a stochastic process and the goal of the agent is determine the optimal time step to stop when the reward to be received is in some sense maximized (Shiryaev] (007). Optimal commitment is unique in that a positive reward can only be received by not stopping until pre-specified time horizon r. In optimal commitment, there is still stochastic process (namely, samples y1) that gradually reveals more information regarding what that positive reward will be at the end, however, the reward- or rather the cost-of of stopping earlier is independent of this stochastic process (and is equal to - -CC).
Sequential hypothesis testing Among other OSPs, optimal commitment most closely related to sequential hypothesis testing (SHT), where an agent makes sequential observations regarding given hypothesis and eventually needs to decide whether to reject the said (alternate) hypothesis or
reject some null hypothesis (Wald and Wolfowitz 1948) Yu et al. 2009; Drugowitsch et al., 2012
Shenoy and Angela 2012 Zhang and Angela Drugowissh et al. 20144) Khalvati and Rao Schönbroh et | 7: Fauu et al. 2020). Rejecting the correct hypothesis provides a positive reward whereas waiting for more observations, while informative, is also costly as in OCP. It is well known that the optimal policy in the classic setting of SHT is thresholding-type policy with fixed thresholds that do not vary over time: The null hypothesis is rejected if some test statistic gets above a threshold (and the alternate hypothesis rejected if the same statistic gets below different threshold).
Optimal commitment can be thought of as SHT problem with the crucial difference that the meta- experimenter has only the option of discarding the alternate hypothesis (i.e. breaking commitment), and once some time horizon is reached (i.e. when a commitment is kept), either the null hypothesis or the alternate hypothesis automatically rejected according to some external success criterion p. regardless of what the meta-experimenterr'' 's decision might have been otherwise. As we have shown in Proposition 2 the optimal policy still remains a thresholding-tyae policy, but since there is now a deadline to discard the alternate hypothesis early, the thresholds become time-varyinge in particular, they become less and less optimistic as the said deadline approaches (cf. Proposition
Frazier and Angela (007)); Dayanik and Angela (2013); y;; Alaa and van der Schaar (2016) consider SHT under stochastic deadlines, but different from optimal commitment, they still allow agents to reject both hypotheses at any time. In these works, the agent must make the rejection decision before the deadline is reached to be able to receive a positive reward, whereas in our case, the agent must wait until the deadline to whether the null hypothesis will be rejected or not. Naghshvar and

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of related experiment designs. Optimal commitment is the only design that aims decide both when an alternative population should be targeted- as opposed to switching the target population only at a fixed decision point-as as well as which population to target among many potential candidates as opposed to simple binary decision of "overall population vs. sub-population" or "go vs. no-go".
Design
Reference
When?
Which?
Randomized Controlled Trial (RCT)
Fisher 1935
Never
Only the initial population
Adaptive Enrichment Design
Ondra al. 2019
Fixed decision point
Overall vs. fixed subpopulation
Adaptive Signature Design
Zhang al 201
Fixed decision point
Possibly any population
RCT with Futility Stopping
He ml. 2012
Possibly any time
Go vs. no-go
Optimal Commitment
(Ours)
Possibly any time Among multiple populations
Javidi (2013); ; Jarrett and van der Schaar 2020) consider active versions of SHT where the agent is able to choose what type of observations to make. Our case is "passive" in the sense that the meta-experimenter cannot influence what kind of samples they are going to receive from the currently running experiment. Finally. optimal commitment, and SHT in general, can be thought of as more structured instances of partially-observe reinforcement learning (RL). As we have discussed earlier, the standard technique here relies on convex reward structures whereas the optimal value function in our case is not convex in general (cf. Proposition
see Appendix for a detailed discussion).
Adaptive experimentation We introduced optimal commitment predominantly as tool for population selection during an experiment. In clinical trials, dominant approach to population selection is adaptive enrichment (Mehta et al. 2009;; Magnusson and Turnbull 2013) Simon and
Simon 2013 Wang and Hung 2013) Simon and Simon 20181000dde et al. 2019 (ha11,2021] and
adaptive signature designs (Freidlin and Simon 2005| Freidlin et al. 2010 Mil 2017/Zhang et al.
2017: Bhattacharyya and Rail (211)). These designs are capable of adapting the target population of trial as the trial continues, but unlike optimal commitment, they can only do so fixed analysis points and not just at any time step. While adaptive signature designs can select arbitrary populations, adaptive enrichment designs are also limited by the number of pre-specified populations they can select between, which is typically only two: the overall population and an alternative subpopulation.
Optimal commitment is also related to clinical trial designs with futility stopping, where an experi- menter might terminate a trial early once it becomes apparent that the said trial is highly unlikely to succeed (vn der Tweel and van Noord 2003 Lachin| 2005 He et al. 2012 Jitlal et al. 2012 Kimani et al. 2013 Chang et al. 2020). However, this does not consider the possibility of switching to new trial that targets different population. As we will see during our experiments, switching to an alternative experiment might prove preferable even before an ongoing experiment can be deemed futile. In such cases, optimal commitment can make more timely decisions. Table 2 summarizes the experiment designs related to optimal commitment. Finally, it is worth mentioning that there are several methods for managing clinical trials at a portfolio level- that is determining which clinical trial is be conducted next (Rogers et al.) 2002; Colvin and Maravelias 2008 Graham et al., 2020). Trial management in this vain is orthogonal to optimal commitment: They are concerned with the success of multiple new treatments and make decisions on trial-by-trial basis whereas we only ever consider a single intervention and make decisions regarding the target population on a sample-by caapple-y-sanne sample basis while experiments still continue. See Appendix H extended related work.
6 EXPERIMENTS
We want to investigate how Bayes-OCP behaves in environments that differ in terms of ground-truth outcomes, for instance, what happens in environments where the original experiment is quite likely to succeed versus what happens in ones where switching to alternative experiment is needed. To this end, we simulate experiments where mean outcomes are varied but other aspects of an experiment fixed: In our environments, there are two atomic- populiaopulationn, X' {KA, ).)) Both atomic- populations have equal propensities 7 122 = 1/2 and the meta-experimeener has the same positively-bians biased prior for the mean outcome associated with each atomic-population: 0x,, Ox (0.0.1, 0.1). Experiment designs targeting one or both of these atomic populatposslaiions all have the same time horizon = 600 and success criterion p(D,) where a = F F-1(95%) So, experiments are powered to detect a positive mean outcome of 0.1 with probability ~ 80. Rewards are given by Rx 1000 the wider the target population is, the more people successful intervention can be marketed to and costs are given by Cx = 1/m2 the narrower the target population is, the harder it becomes to find subjects eligible to participate.
{( y/ |Prl a/ VF)
7

---

Published as a conference paper at ICLR 2023
Benchmarks We consider the meta- experiment designs summarized in Table 2 as benchmarks (see Appen- dix A. for an RI.-based benchmark). Conventional RCT always targets the overall population and never stops carly-that is it always conducts the experiment u = (AA,.A)) T. P. un- til its completion. Adaptive Enrich- ment performs an intermediary anal-
Table 3: Performance comparison in various environment in- stances. Bayes- has the highest expected utility- and a smaller FWER then conventional RCTs- averaged over all environ- ment instances. This is because Bayes-OCP a balanced design whose structure does not favor certain environment instances over others. As an example, compare with conventional RCTs: RCTs do not have an adaptive structure hence they favor green environ- ments where it is not necessary to adapt the target population of the initial experiment. IInstances favored/addressed partially
Algacilees:
Oual
RCT
Adatir
Funils Skezpas
Cuerdy
Beyes-OCP
ysis at L = =/2 = 300 and greed- ily selects the experiment with the highest expected utility from v = x))xx((x Futility Stop- ping is implemented via Bayes-OCP by initializing the set of all experi- ments as a singleton V v ([KA, Kn), T, p)}. Intuitively, futil- ity stopping only decides whether or not stop the initial experiment that targets the overall population early. Bayes-OC' is initialized with / = 0.80 (see Appendix E for a sensitivity analysis). We also consider an abla- tion of Bayes-OCP where decisions are made greedily instead of optimistically (Greedy Bayes- As a baseline of maximum achievable performance, we consider an oracle (Oracle RCT) that always runs the RCT with the optimum target (or does not run any RCT at all if that happens to be optimal).
Favored Inotanee
NDA
Cuem
Grcesflanbe
GroadRee
Andsa Mal
(teck Aithert
Bolanced
Enitar
2011
-74. (5-7)
195 (5.9)
(2.09
12.11111
(71
All
FWER
00%
0.55 (0L1R)
0.25 (2.1))
6.1% (0.)%)
CUP% (00%:
0.14 (49)
Inatescc
Swildeo
(D00)
(0:50
1.0 (005
0.600.0
(0.79)
58 26
45,4%
26)
Ton-5
600 (00)
0038/20)
6000 (8:05
607.51144
615 11 7g
Tem-ll
400/1
9/69/1611
14:00:1
Uniltt
569
386.7
(356
357.7 65-77
63.1
4754 5)
Green
FWER
0.6%
055 (005)
0 (3055)
6.0% (0.03
P9 (008)
0.0% (9 (5)
Instencca
Seridan
CUT (50))
00/0000
(8.00
099 (U0)
10:0
00LD
6038 (50)
00301200 (20))
6560 (8.00
605.811.51
6G2.80300 (3.4)
TroFF
0000
4000
759 496
(7.6
2.5
(145
Unitty
258.
5003
(7:5555
(5:1
Audrer
PWER
0.%
0.7% (0.3)
056 (033))
6 (034)
6.9 (00%
0.3% (9 (5)
Instanoes
Sentches
09:00
14m-
59311 (00)
TwuF
0000
69000 (00)
1450
16.3 00.3)
3.4
(0.4
Red
PWER
00%
0.2% (6.3))
0.2% (239)
0.1% 00290
CAPR (MDR)
0.2% (2.99)
Swildas
1.0
SUR (0U0)
L0:20
1,0 10.00
1.1 (061
1.000:0
Tin-5
400/1
600
6000
691 (1
Ten-F
0.0
400 (0U0)
(4.1
(-5
Environments A meta--xperrimenter performance is specific to the environment instance. In particular, it depends on the ground-truth outcome distributions {Ω₂} } for different populations. For example, an algorithm that always immediately stops the experiment would perform best when the mean outcome is negative. Hence, to faithfully evaluate the benchmarks, we need to focus on the average performance across different environments. To this end, we randomly generated 1000 environments (repeated five times to obtain error bars) with true mean outcomes 0x,, Ox2 sampled independently from (0..1, 0.1). Given these means, outcome distributions are set to be Gaussian with unit variance such that Ω22 = (N20, 1). Depending on the true mean outcome, these environments can be categorized into three groups: (i) green instances where the initial experiment targeting the overall population has the highest utility, (ii) amber instances where an alternative experiment that targets a subpopulation has the highest utility, and (ii) red instances where no experiment has positive utility hence running no experiments is the optimal decision.
Different benchmarks favor different instances (see the top row of Table Conventional RCTs do not allow for any adaptation hence they favor green instances where the target population of the initial ex- periment does not need to be adapted. Adaptive Enrichment allows for adaptation but only at certain time point, which is often too late to stop unsuccessful experiments (as in red instances). However, an adaptive enrichment design at least makes it possible eventually target subpopulation, even though it might be too late to do so at the pre-specified decision point, hence it partially accommodates amber instances. Futility Stopping decides between either continuing with the initial experiment or stopping all experimentation completely (targeting a subpopulation is not an option) hence it favors either green or red instances (but not amber instances). Greedy Bayes- OCP is pessimistic (or rather not optimistic enough) towards any ongoing experiment succeeding, hence it favors red instances where no experiment is likely to succeed. Similar adaptive enrichment, Greedy Bayes-OCr at least allows subpopulations to be targeted hence it too partially accommodates amber instances.
Main results Performance of a meta-experimenter is primarily measured by Bayesian utility which is the expected utility averaged over randomly sampled environment instances (Utility). Remember that maximizing utility was our main objective, and as such, Bayes-OCP has the highest expected util- ity when averaged over all environment instances, see Table 3 Unlike other benchmarks, Bayes-OCP strikes a good balance in prioritizing all environment instances at the same time. This is because Bayes-
8

---

Published as a conference paper at ICLR 2023
OCP (i) can make timely decisions-unlie Adaptive Enrichment- and (ii) is optimistic hence it does not stop sikely-to-succeee experiments prematurely-unlike Greedy Bayes-OCP. More specifically,
toports Si
To aecuele
(i) Timeliness of Bayes-OCP: Bayes-OCP has an ad- vantage in amber and red instances over adaptive enrichment and futility stopping. Consider the exam- ple in Figure 3 While Bayes-OCP stops in timely manner, adaptive enrichment can only stop at fixed decision point and experiments with futility stopping only stop when the ongoing experiment is failing not soon as a better alternative emerges. This under- lines the exploitative aspect of Bayes-OCP makiin and breaking commitments to maximize utility.
Sanddking aod
Latoocia Best Allippte
Figure 3: Timeliness of Bayes-OCP. Bayes- OCP is first to (correctly) stop the initial ex- periment in amber instance (excluding Greedy Bayes-OCP). Adaptive enrichment can only stop at pre- pre-specified time, while futility stopping fails to consider switching to alternative experiment, which is proven be preferable earlier than stopping.
Optimism of Baye While a design that fa- vors early stopping is obviously desirable in amber and red environments, how much it is favored should be moderated to also succeed in green environments, Consider the example in Figure 4 Greedy Bayes- OCP prematurely stops the initial experiment in green environment while Bayes-OCP does not. The- oretically, we know that the optimal policy should be optimistic towards the ongoing experiment suc- ceeding and be hesitant to stop to a certain extend. This underlines the exploratory aspect of Bayes- OCP-keeping a seemingly failing commitment still has value as it reveals more information regarding
whether the commitment is actually failing.
In Table in addition to Utility, we also report the family- wise error rate (FWER)- is the frequency of runs where at least one experiment (denote it with w/") is de- clared successful (i.e. (D)) = 1) despite the mean outcome being negative for the targeted population (i.e. 0x. < the average number of times the target pop- ulation has been switched (Switches), the probability of success which is defined as achieving positive utility (Suc- cess), the average time until successful outcome (Time- to-Success, T-to-S), and the average time until until an un- successful outcome where all experimentation is stopped with negative utility Timeetto-Fillure T-to-F), see Appendix G for details. Importantly, Bayes-OCP does not compromise the error control of experiments, on the contrary, even achieves a smaller FWER than conventional RCTs. This is because aggregate data is only ever used to select experiments, otherwise no two experiments consult each other's 's data when evaluating a success criterion so that the potential confoundedness that could have been caused by the adaptiveness of Bayes- OCP is avoided when declaring an experiment as successful (see Appendix for a discussion on error control).
Figure 4: Optimism of OCyes-OCP. Greedy Bayes-OCP (incorrectly) stops due to initial noise a green instance while Bayes-OCP does not stop since it is more optimistic (as the optimal policy should, cf. Proposition 3
Supplementary results
We also provide supplementary results: Appendix eevealuuusseesl evaluates RL-based error control, Appendixx[oconiierrs environments with non-
benchmarks, (rreendix...iinvsttieee Gaussian outcomes, Appendix conniddess environments with more than two atomic-populations, and Appendix F analyzes the sensitivity of Bayes-OCP's performance to its hyper-parameter B.
7 CONCLUSION
Two aspects of OCP require further discussion: (i) How can it be approached from the perspective of reinforcement learning? While OCP technically describes a special class of POMDPs, we have not found this to be constructive in finding solution (see Appendix Ap. (ii) What are the implications of using Bayes-OCP in terms of error control? It has no impact on individual error rates and can be adapted control FWER (see Appendix B See Appendix FF for discussion future work.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
As the main application of optimal commitment, we have focused on adaptive experimentation, particularly experiments that are run as part of clinical development. Clinical trials have huge impact the wellbeing of patients and this high-stakes nature of clinical trials naturally raises some ethical concerns; we discuss two major ones in this section. However before we start our discussion, it should be emphasized that clinical trials is not the only application domain of optimal commitment. As we have highlighted at the end of Section 2 our contributions are generally applicable to decision- making problems such portfolio and energy management, Moreover, not all adaptive experiments are clinical and have the same high stakes as a clinical trial. For instance, A/B testing is common in online advertisement to determine what recommendation policies lead to more user engagement (Gui et al. 2015 Xu et al. 2015: Kohavi and Longbotham (0117). Therefore, the ethical concerns we discuss here does not universally concern all possible applications of optimal commitment.
The first concern is how the designed eπror rate of an individual experiment is affected when multiple such experiments are managed together using Bayes-OCI OCP in an adaptive manner, in particular, whether any error rate is inflated by the use of Bayes-OCP or not. We discuss eттоr control in Appendix [ with supplementary experiments. But briefly, Bayes-OCP has essentially no impact on the error rate of experiments on an individual level, and when controlling their family-wise error rate is also a concern, it can easily be adapted to accommodate this additional constraint as well.
The second concern is that an adaptive approach to population selection might lead to overly conser- vative experiments that unnecessarily limit the use of an effective treatment. As we have mentioned in the introduction to motivate the need for optimal commitment, when the treatment is effective only for a subpopulation (cf. amber instances in our experiments), population selection is absolutely necessary, otherwise the treatment is most likely to be found ineffective and discarded after an experiment that targets the overall patient population as a whole, which would deny the treatment for the subpopulation that would have benefited from it. On the flip side of this, when the treatment happens to be effective for everyone (cf. green instances in our experiment), population selection might lead to conducting restrictive experiment that only targets a small subpopulation, which this time, would deny the treatment for the rest of the patient population. This is essentially the reason behind the performance drop between Bayes- Bayes-OCP and conventional RCTs in green instances (see Table There is a trade-off between the performance in amber instances and green instances; and Bayes-OCP achieves a better balance between the two compared with a conventional RCT as evidenced by its superior performance when averaged over all environment instances (again see Table although it causes a drop in performance for green instances, it more than makes up for that drop amber instances. This balance is partly controlled by how optimistic Bayes-OCP is, which 3-larger 3 leads to more optimistic decisions towards
is in turn dictated by its hyper-paramette ongoing experiments, which favors green instances more then amber instances. We analyze the sensitivity of Bayes-OCP's performance to 3 in Appendix D.] and for all configurations that we have evaluated, Bayes-OCP always performs significantly better than a conventional RCT.
REPRODUCIBILITY STATEMENT
All our experiments are based on synthetic simulations, hence our results can easily be reproduced by following the specifications in Section 6 without needing access to any private dataset. In order to aid reproducibility. we have rigorously described all our benchmarks in algorithmic form, similar to Algorithm I in Appendix D Moreover, the source code necessary to reproduce our main results in Table is made publicly available at https: wwt.:hhh...oooccccoooomooooo...e a hhwnihhaanooooo /otttcotttommmomoo and https: wwtp///aaeeeeescaaaaaaaa/agoegga
ACKNOWLEDGMENTE
We would like to thank the reviewers and the members of the van der Schaar lab, for their valuable input, comments, and suggestions. This work was supported by the US Office of Naval Research (ONR) and the National Science Foundation (NSF, grant number 1722516).
REFERENCES
Alaa, A. M. and van der Schaar, M., "Balancing suspense and surprise: Timely decision making with endogenous information acquisition," in Proc. Neural Inf. Process. Syst., 2016.
10

---

Published as a conference paper at ICLR 2023
Auer, P., Mach. Learn., vol. 47, no. 2, pp. 235-256, 2002.
N., and Fischer, P., "Finite- time analysis of the multiarmed bandit problem."
Bhattacharyya, A. and Rai, S. N., "Adaptive signature design-revieey of the biomarker guided adaptive phase-Ill controlled design." Contemporary Clin. Trials Commun., vol. 15, p. 100378, 2019.
Bubeck, S., BiiacBianchi, N. et al., "Regret analysis of stochastic and nonstochastic bandit problems," Found. Trends Mach. Learn., vol. 5, no. 1, pp. 1-122, 2012.
Chang, Y., Song, T., Monaco, J., and Ivanova, A., "Futility stopping in clinical trials, optimality and practical considerations," J. Biopharmacentical Statist., vol. 30, no. 6, pp. 1050-1059, 2020.
Chiu, Y.D.., Koenig, F., Posch, M., and Jaki, T., "Design and estimation clinical trials with subpopulation selection," Statist. Med., vol. 37, pp. 4335-4335-4352, 2018.
"Trends, charts, and maps," ClinicalTrials.gov. [Online]. Available: https:/wwww.clliccariiaa.sssp Rtt/ss/wwsssoooooeeeeeeeeooieooe
Colvin, M. and Maravelias, C. T., "A stochastic programming approach for clinical trial planning in new drug development," Comput. Chem. Eng., vol. 32, no. 11, pp. 2626-2642, 2008.
Dayanik, S. and Angela, J. Y., "Reward-rate maximization in sequential identification under stochastic deadline," " SIAM J. Control Optim., vol. 51, no. 4, pp. 1922-2948, 2013.
Demets, D. L. and Lan, K. K. G., "Interim analysis: The alpha spending function approach," Statist. Med., vol. 13, no. 13-14, pp. 1341-1352, 1994.
Drugowitsch, J., Moreno-Bote, R., Churchland, A. K., Shadlen, M. N., and Pouget, A., "The cost of accumulating evidence in perceptual decision making," J. Neuroscience, vol. 32, no. 11, pp. 3612-3628, 2012.
Drugowitsch, J., Moreno-Bote, R., and Pouget, A., "Optimal decision-making with time-varying evidence reliability," in Proc. Neural Inf. Process. Syst., 2014.
FauB, M., Zoubir, A. M., and Poor, H. V., "Minimax optimal sequential hypothesis tests for Markoy processes," Ann. Statist,, vol. 48, pp. 2599-26621 2020.
Fisher, A., The Design of Experiments. Edinburgh, Scotland: Oliver & Boyd, 1935.
Frazier, P. and Angela, J. Y., "Sequential hypothesis testing under stochastic deadlines," in Proc. Neural Inf. Process. Syst., 2007.
Freidlin, B. and Simon, R., "Adaptive signature design: An adaptive clinical trial design for generating and prospectively testing a gene expression signature for sensitive patients," Clin. Cancer Res., vol. 11, no. 21, pp. 7872-7878, 2005.
Freidlin, B., Jiang. W., and Simon, R., "The cross-validated adaptive signature design," Clin. Cancer Res., vol. 16, no. 2. pp. 691-698. 2010.
Ghare, G. and Leutenegger, S. T., "Improving speedup and response times by replicating parallel programs SNOW," in Proc. Int. Conf. Job Scheduling Strategies Parallel Process., 2005.
Graham, E., Jaki, T., and Harbron, C., "A comparison of stochastic programming methods for portfolio level decision-making, J. Biopharmaceutical Statist., vol. 30, no. 3, pp. 405,429, 2020.
Gui, H., Xu, Y., Bhasin, A., and Han, J., "Network A/B testing: From sampling to estimation." in Proc. Int. Conf. World Wide Web, 2015.
Hc, P., Lai, T. L., and Liao, O. Y.W.., "Futility stopping in clinical trials." " Statist. Interface, vol. 5, no. 4, pp. 415-423, 2012.
Jarrett, D. and van der Schaar, M., "Inverse active sensing: Modeling and understanding timely decision-making." in Int. Conf. Mach. Learn., 2020.

---

Published as a conference paper at ICLR 2023
Jitlal, M., Khan, I., Lee, S., and Hackshaw, A., "Stopping clinical trials early for futility: retrospective analysis of several randomised climical studies," Brit. J. Cancer, vol. 107, no. 6, pp. 910-917, 2012.
Kaitin, K. I., "Deconstructing the drug development process: The new face of innovation," Clin. Pharmacology Therapeutics, vol. 87, no. 3, pp. 356-33 356-361, 2010.
Karatzas, I. and Wang, H., "Utility maximization with discretionary stopping," SIAM J. Control Optim., vol. 39, pp. 306-329 2020.
Khalvati, and Rao, P. "A Bayesian framework for modeling confidence perceptual decision making," in Proc. Neural Inf. Process. Syst., 2015.
Kimani, P. K., Todd, S., and Stallard, N., "Conditionally unbiased estimation in phase II/III clinical trials with early stopping for futility," Statist. Med., vol. 32, no. 17, pp. 2893-2910, 2013.
Kohavi, R. and Longbotham, R., "Online controlled experiments and A/B testing," Encyclopedia Mach. Learn. Data Mining, vol. 7, no. 8, pp. 922-929, 2017.
Lachin, J. M., "A review of methods for futility stopping based on conditional power," Statist. Med., vol. 24, no. 18, pp. 2747-2764, 2005.
Lawler, E., Combinatorial Optimization, Networks and Matroids. New York: Holt, Rinehard Winston, 1976.
Lipkovich, I., Dmitrienko, A., and D' Agostino Sr., B., "Tutorial biostatisties: Data-driven subgroup identification and analysis in clinical trials," Statist. Med., vol. 36, pp. 136-196, 2017.
Magnusson, B. P. and Turnbull, B. W., "Group sequential enrichment design incorporating subgroup selection," " Statist. Med., vol. 32, no. 16, pp. 2695-2714, 2013.
Markowitz, H., Portfolio selection: Efficient diversification of investment. New York: John Wiley, 1959.
Mehta, C., Gao, P., Bhatt, D. L., Harrington, R. A., Skerjanec, S., and Ware, J. H., "Optimizing trial design: Sequential, adaptive, and enrichment strategies," Circulation, vol. 119, no. 4, pp. 597-605, 2009.
Merton, R. C., "Liie time portfolio selection under uncertainty: The continuous-time case," Rev. Econ. Statist., vol. 51, pp. 247-257, 1969.
Mi, G., "EEhancement of the adaptive signature design for learning and confirming in a single pivotal trial." Pharmaceutical Statist., vol. 16, no. 5, pp. 312-321, 2017.
Moineddin, R., Butt, D. A., Tomlinson, G., and Beyene, J., "Identifying subpopulations for subgroup analysis in longitudinal clinical trial," Contemporary Clin. Trials, vol. 29, pp. 817-822, 2008.
Naghshvar, M. and Javidi, T., "Active sequential hypothesis testing," Ann. Statist., vol. 41, no. 6, pp. 2703-2738, 2013.
Ni, T., Eysenbach, B., and Salakhutdinov, R., "Recurrent model-free RL can be strong baseline for many POMDPs," in Proc. Int. Conf. Mach. Learn., 2022.
Olofsson, M., Önskog, T., and Lundström, N. L. P., "Management strategies for run-of-river hy- dropower plants: An optimal switching approach." " Optim. Eng., vol. 23, pp. 1707-1731, 2022.
Ondra, T., Jobjörnsson, S., Beckman, R. A., Burman, C.-F., König, F., Stallard, N., and Posch, M.,
"Optimized adaptive enrichment designs," Statist. Methods Med. Res., vol. 28, no. 7, pp. 2096-2111, 2019.
Papadimitriou, C. H. and Steiglitz, K., Combinatorial Optimization Oprimization-AAgoriimee and Complexity. Englewood Cliffs, NJ: Prentice Hall, 1982.
Rafique, S. F. and Jianhua, Z., "Energy management system, generation and demand predictors: A review"" IET Gener. Transmiss. Distribution, vol. 12, no. 3, pp. 519-530, 2018.
12

---

Published as a conference paper at ICLR 2023
Rogers, M. J., Gupta, A., and Maranas, C. D., "Real options based analysis of optimal pharmaccutical research and development portfolios," " Ind. Eng. Chem. Res., vol. 41, no. 25, pp. 6607-6620, 2002.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., and Perugini, M., "Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences," Psychol. Methods, vol. 22, no. p. 322, 2017.
Shenoy, P. and Angela, J. Y., "Strategic impatience in Go/NoGo versus forced-choice decision- making," in Proc. Neural Inf. Process Syst., 2012.
Shiryaev, A. N., Optimal Stopping Rules. Springer Science Business Media, 2007.
Simon, N. and Simon, R., "Adaptive enrichment designs for clinical trials," Biostatistics, vol. 14, pp. 613-625, 2013.
Simon, N. and Simon, R., "Using Bayesian modeling in frequentist adaptive enrichment designs." Biostatisticss vol. 19, no. pp. 27-41, 2018.
Spaan, M. T. J., "Partially observable Markov decision processes," in Reinforcement Learning. Springer, 2012, pp. 387-414.
Takehe, T., imai, R., and Ono, S., "The current status of drug discovery and development as originated in United States academia: The influence of industrial and academic collaboration drug discovery and development," Clin. Transl. Sci., vol. 11, no. 6, pp. 597-606, 2018.
Thall, P. F,, "Adaptive enrichment designs in clinical trials," Annu. Rev. Statist. Appl., vol. 8, pp. 393-411, 2021.
Umscheid, C. A., Margolis, D. J., and Grossman, C. E., "Key concepts of clinical trials: A narrative review," Postgraduate Med., vol. 123, no. 5, pp. 194-204, 2011.
van der Tweel, I. and van Noord, P. A., "Early stopping in clinical trials and epidemiologic studies for 'futility": Conditional power versus sequential analysis," J. Clin. Epidemiology, vol. 56, no. 7, pp. 610-617, 2003.
Wald, A. and Wolfowitz, J., "Optimum character of the sequential probability ratio test," Ann. Math. Statist,, vol. 19, pp. 326-339, 1948.
Wand, D., Joshi, G., and Wornell, G., "Efficient task replication for fast response times in parallel computation," " in Proc. ACM SIGMETRICS Conf., 2014.
Wang, D., Joshi, G., and Wornell, G. W., "Efficient straggler replication in large-scale parallel computing." ACM Trans. Model. Perform. Eval. Comput. Syst., vol. 4, no. 2, pp- 1-23, 2019.
Wang, S.-J. and Hung, J., "Adaptive enrichment with subpopulation selection at interim: Method- ologies, applications and design considerations," Contemporary clinical trials, vol. 36, no. 2, pp. 673-681, 2013.
Xu, Y., Chen, N., Fernandez, A., Sinno, O., and Bhasin, A., "From infrastructure to culture: A/B testing challenges in large scale social networks," in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2015.
Yu, A. J., Dayan, P., and Cohen, J. D., "Dynamics of attentional selection under conflict: Toward a rational Bayesian account," J. Exp. Psychol. Human Perception Performance, vol. 35, no. 3, p. 700, 2009.
Zhang, S. and Angela, J. Y., "Forgetful Bayes and myopic planning: Human learning and decision- making a bandit setting," " in Proc. Neural Inf. Process. Syst., 2013.
Zhang, Z., Li. M., Lin, M., Soon, G., Greene, T., and Shen, C., "Subgroup selection in adaptive signature designs of confirmatory clinical trials," J. Roy. Statist. Soc.: Ser. C (Appl. Statist.), vol. 66 no. 2, pp. 345-361, 2017.
13

---

Published as a conference paper at ICLR 2023
A A REINFORCEMENT LEARNING PERSPECTIVE
Optimal commitment can be viewed as partially-observed reinforcement learning problem. Let the tuple A, T,
O O,R) R) denote a partially- partially-observable Markov decision process (POMDP), where
S is the (unobserved) state space, A is the action space, Z is the observation space, T E A(S)SxS describes the transition dynamics,
E g describes the observation dynamics, and describes the reward dynamics. Then, OCPs defined in Section S can also be expressed class of POMDPs: Letting y = R denote the outcome space for clarity, D = Ui=0(A' x 3) be the space of all possible datasets Dr, and D be the space of all possible outcome distributions Ω
RS special
S {Ø) U ( x D x D² where states 8 = (1/, Dt, {Rtr consist the ongoing experiment & V, the dataset D, D collected by the ongoing experiment so far, and the true outcome distributions {St, E D),
A (Ø) V,
= (0) U (X' )),
T[(s 0,) = and
T(s (vb (Q.)))
0
if
8' = (b,,D+1= = D ( (1x))
Sd.. I++1 {[=(p] 3t+1 Sreess
af==v
8 (w', Di (.,,8) (
s.t. σ ~ ( ~ Ori
if = v' = (X',
O(s = Ø) = 0 and
O = (bb, Di+1 D, U (Ic-1 M411). (12))) = (2141.8641),
R(s' 13) = and
R(8's = (w (X, De,1, (Rx))) Cwp + Ry: 1[(+1= (((+4))
Since ongoing experiments vb are completely dictated by actions, and datasets D, collected by the ongoing experiments consist solely of observations (I2, Wt), the only unobserved component of the states in this POMDP is the true outcome distributions {2z (22.. Hence, the optimal policy should have the form x (अ b) where E A(D*)) denotes beliefs over {9t )-tt)t is posterior
distributions over the true outcome distributions. For instance, when Qg = NO 1) as we have been assuming in Sections 3 and 4 posteriors
mean outcomes {0z) which are given by N o2), constitute as beliefs.
parameters (Ma, o²) such that 0g
Now although an OCP be expressed as a POMDP, doing is not particularly helpful in finding the standard approach to solving a POMDP would be dynamic programming and compute the optimal value function V* and the optimal Q-function Q' iteratively according to Bellman optimality conditions
a solution. As we have already discussed in Section 3,
Q" (b, a) Earh..' (0t [R(s') (R
(*(b)= = max.EA Q*
where 8' 2'} denotes the updated belief b' after having belief b and making a new observa- tion 2'. When the state space discrete or equivalently in our case, when the space of outcome distributions Ω E D is discrete--' and Q" happen to be convex functions, which makes it pos- sible to perform these iterations efficiently by approximating V* and Q* using functions of the form f(b) max(a;b a (Spaan) 2012). However, even in the simplest of cases where S is continuous or equivalently, the space of outcome distributions E D is continuous, for instance when Ω2 AV 1) 1)-ttheeeeee the convexity of V* and Q' no longer generally holds. In fact, show in Proposition that neither V nor - V is convex with respect to beliefs = (t, p1} for at least one instance of the simplified OCP that have analyzed in Section
14

---

Published as a conference paper at ICLR 2023
Table 4: Performance comparison between Futility Stopping with RL-based algorithms and with Bayes-
Oracle RCT
Futility Stopping
Furility Stopping w/ Deep Q-learning
Futility Stopping w/ Bayes-OCP
Algorithms:
w/ Discretized RI.
Utility
260.4
131.8 (4.3)
78.8 (3.1)
150.0 (3.5)
All
FWER
0.0%
0.1% (0.1%)
0.0% (0.0%)
0.1% (0. (0.1%)
Instances
Switches
0.5
0.6 (0.0)
0.7 (0.0)
0.5 (0.0)
(100%)
Success
75.2%
41.0% (1.0%)
24.2% (0.8%)
45.4% (1.3%)
T-to-S
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
T-to-
35.6
54.3 (2.2)
23.6 (1.8
57.6 (4.6)
Utility
389.6
309.5 (4.1)
185.0 (4.9)
337.7 (5.7)
Green
FWER
0.0%
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
Instances (47.3%)
Switches
0.0
0.2 (0.0)
0.5 (0.0)
0.1 (0.0)
Success
99.0%
80.9% (0.7%)
47.7% (1.1%)
86.0% (1.4%)
T-to-S
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
T-to-F
600.0
72.7 (7.1)
11.0 (0.8)
46.6 (7.6)
Utility
258.6
23.99 (5.4)
-16.6 (6.8)
.5.3 (5.4)
Amber Instances (29.4%)
FWER
0.0%
0.2% (0.2%)
0.1% (0.1%)
0.4% (0.3%)
Switches
1.0
0.9 (0.0)
0,9 (0.0)
0,8 (0.0)
Success
96.6%
9.1% (0.8%)
5.2% (1.3%)
15.2% (2.0%)
T-to-S
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
T-to-F
600.0
66. (4.3)
39.5 (4.4)
78.3 (9.3)
Utility
0.0
-3.00 (1.6)
-16.5 (5.2)
-35.1 (1.7)
Red Instances (23.3%)
FWER
0.0%
0.1% (0.2%)
0.1% (0.2%)
0.1% (0.2%)
Switches
1.0
1.0 (0.0)
1.0 (0.0)
1.0 (0.0)
Success
0.0%
0.2% (0.2%)
0.4% (0.4%)
0.9% (0.3%)
T-to-S
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
T-to-F
0.0
33.7 (0.9)
18.3 (4.8)
38.9 (2.1)
A.1 EXPERIMENTS WITH REINFORCEMENT LEARNING BENCHMARKS
Having said all that, one naive way to still compute V and Q* iteratively according to Bellman optimality conditions is to discretize the belief space. We call this benchmark Discretized RL and we use to perform futility stopping- viable experiment design early not. Otherwise, the dimensionality of the belief state explodes combinatorially with respect to V We consider the same setting that we have considered during our experiments in Section and compare the performance of Futility Stopping with Discretized RI. with that of Futility Stopping with Bayes-OCP. When implementing discretized RI., instead of keeping track of the entire dataset D,, we only keep track of the sufficient statistic 14, E( (.) Be We/l1 [Dr), restrict the domain of Pr to interval [-0.3, 0.3), and discretize this interval into 100 equally-spaced bins. In addition to discretized RL, we also consider the approach proposed by Ni et al. (2022) for solving complex classes of POMDPs, which the optimal commitment problem is one of. Briefly, employ deep Q-learning (as such, we call this benchmark Deep Q-learning) network as an approximation of the Q-function Q" (b, a) using the POMDP we formalized earlier as simulator. As the network architecture, we consider multi-layer perceptron with two hidden layers of size 100 and with tanh activations.
is when | 1, deciding whether to stop the only
train neural
Results are given in Table 4 futility stopping with Bayes-OCK performs better than futility stop- ping with discretized RL as well as futility stopping with deep Q-learning. In addition to the bad performance of discretized RL, it is also not feasible scale it to use cases beyond futility stopping. When / > 1. we would need to keep separate track of each H4, Moreover, we would also need to start keeping track of the scale parameters {az} since would now possible to distribute samples among multiple atomic-pon coamiicpopulations in multiple ways by targeting different populations with different experiments (we no longer would be able to treat the target population of the only viable experiment design as the only atomic-population there is). Noting that o='s 's already take discrete values with at least 7 7-many possible values, merely increasing the number viable experiments | from one to two causes the dimensionality of the belief space to jump from 100 to (100 600) 36 1 Deep Q-learning performs even worse as it ignores all structure present in the optimal commitment problem, and instead, views the POMDP that describes it as a black-box simulator.
15

---

Published as a conference paper at ICLR 2023
Table 5: Performance comparison of algorithms with family-wise error control.
Onacle
Adaptive Farichment
Futility Stogping w/ Bayes- OCP
Gresdy Bayes-OCP w/ Bonferromi Corr.
Bayes-OCP
Algorithms:
RCT
RCT
w/ Bonferroni Corr.
w/ Bonferroni Corr.
Uuiiiy
260.4
-3..4
(5
(3.5)
23.7 (2.2)
158.7 (5.2)
All
FWER
0.0%
0.3% (0.1%)
0.1% (0.1%)
0.1% (0.1%)
0.0% (0.0%)
0.1% (0.1%)
Insfances
Switches
(0.0)
0.5 (0.0)
0.5 (0.0)
(0.0)
(0.0)
(100%)
Success
75.2%
56.1% (0.7%)
51.1% (0.7%)
45,4% (1.3%)
7.7% (0.5%)
49.3% (1.5%)
T-to-S
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
606.2 (2.3)
616.6 (1.7)
Tio-F
35.6
600.0 (0.0)
543.0 (15.7)
57.6 (4.6)
2.3 (0.3)
65.8 (7.0)
Unlity
(3.9)
378.8 1)
337.7 (5.7)
46.1 (3.3)
325.8
Green
FWER
0.0%
0.0% (0.0%)
0.09% (0.0%)
0.09% (0.05%
0.0% (0.0%)
0,0% (0.0%)
Instances
Switches
(0.0)
0.0 (0.0)
0.1 (0.0)
(0.0)
0.2 (0.0)
(47.3%)
Success
99.0%
98.9% (0.4%)
96. 1% (0.6%)
86.0% (1.4%)
13.8% (0.8%)
84.7% (1.5 (1.5%)
T-to-S
600.0
600.0
600.0 (0.0)
600.0 (0.0)
604.9
(0.7)
T-to-F
600.0
600.0
(19.6)19.0)
46.6 (7.6)
2.0 (0.3)
66.5 (17.7)
Uniliy
258.6
.300.3
(15.6)
(5.4)
8.3 (2.9)
44.6 (4.6)
Amber
PWER
0.0%
0.7% (0.3%
0.3% (0. (0.)
0.4%: (0.3%%
0.0% (0.0%%
0.2% (0.25%
Insfances
Switches
(0.0)
0.8 (0.0)
0.8 (0.0)
(0.0)
(0.0)
(29.4%)
Success
96.6%
30.0% (2.0%)
18.2% (2.2%)
15.2% (2.0%)
3,8% (0.9%)
30.7% (1.4%)
T-to-S
600.0
600.0
600.0 (0.0)
600.0 (0.0)
613.4
670.6 (8.9)
T-io-F
600.0
600.0 (0.0)
724.6 (9:4)
78.3 (9.3)
2.6 (0.6)
95.5 (18:4)
Uiility
.579.2
(2.3)
-35.1 (1.7)
-2.2 (0.3)
-37.0 (2.4)
Red
FWER
0.0%
0.2% (0.3%)
0.2% (0.3%)
0.1% (0.2%)
0.0% (0.0%)
0.1% (0.2%)
Instances (23.3%)
Switches
(0.0)
1.0 (0.0)
1.0 (0.0)
(0.0)
(0.0)
Success
0.0%
2.1% (0.4%%
1.1% (0.44%
0.9%% (0.35)
0.1% (0.2%)
1.1% (0.7%)
T-to-S
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
649.8 (62.6)
T-to-F
0.0
600.0 (0.0)
334.9 (3.7)
38.9 (2.1)
2.4 (0.5)
39.8 (2.5)
B DISCUSSION ON ERROR CONTROL
Bayes-OCP is method for managing experiments- that is deciding what experiment to conduct and when- as opposed to a hypothesis testing strategy in and of itself. Implication of this in terms of error control is that the type error of any individual experiment run by Bayes-OCP can always be controlled by choosing an appropriate experimental design, in particular, by specifying an appropriate success criterion p. This individual-level error control built into the design of each experiment is not compromised by Bayes-OCP; no aggregate data from multiple experiments is ever fed into the success criterion of one alone (see Section 2 experiment s is successful if p² (D)) p' pD(:) 1); and any assumptions made by Bayes-OCP regarding outcomes in Section whether accurate or inaccurate, have no effect on the results produced by an external success criterion.
not if
While Bayes-OCP does not compromise the individual error control of experiments, neither does it control their collective family-wist error rate (FWER)-thee that the probability of at least one experiment among all that are conducted making false discovery. Bayes-OCP views the problem of managing experiments purely as a utility maximization problem with no additional constraints. Within the scope of our discussion, the purpose of measuring FWER as metric is to check empiri- cally whether the individual error rates are inflated or not (note that FWER a stricter notion of error than individual error rate). In practice, depending on how closely related the managed experiments are, controlling FWER might not necessarily be a concern. Let us highlight this: Any algorithm that manages experiments for long enough is bound to make at least one false discovery. Each year more than a thousand clinical trials are launched (that eventually post results) and more than half of these trials succeed (akebe al. 2018 Cli). If the type I error rate of all these trials were %5, we would expect at least 25 false discoveries in a year, which is more than one hence it would have put FWER of all real- world trials at almost 100% when measured in year- bear-by-year basis. Of course, this not prob- lematic since not all clinical trials are related to each other closely enough to be considered family.
B.1 EXPERIMENTS WITH FAMILY- ERROR CONTROI.
When controlling FWER is of concern, Bayes-OCP can easily be adapted to satisfy this additional constraint by first limiting the number of total experiments that can be conducted- is putting an upper bound on æ-annd then using well-established methods for family-wise error control such as Bonferroni correction or alpha spending functions (Demets and Lan 1994) to adjust the success criteria of the viable experiments in V. We run additional experiments to evaluate the performance of Bayes-OCP -CP with Bonferroni correction. We consider the same setting that we have considered during our experiments in Section excett for one difference: We limit the number of experiments
16

---

Published as a conference paper at ICLR 2023
Table 6: Performance comparison when the ground ground-truth truth outcome distributions not Gaussian.
Oracle RCT
Adaptive Fnrichment
Futility Stopping
Greedy Bayes-OCP
Algorithms:
RCT
w/ Bayes- OCP
Bayes-OCP
Utility
266.5
-38.4 (14.7)
110.0 (9.5)
150.8 (8.5)
46.3 (3.8)
178.2 (7.3)
All Instances
FWER
0.0%
0.1% (0.1%)
0.0% (0.1%)
0.0% (0.1%)
0.0% (0.0%)
0.0% (0.1%)
Switches
(0.0)
(0.0)
(0.0)
(0.0)
(0.0)
(100%)
Success
76.6%
56.2% (1.5%)
53.6% (1.5%)
46.5% (1.8%)
14.7% (1.1%)
54.9% (1.7%)
T-t-S
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
607.3 (1.0)
617.2 (1.8)
T-to-F
32.5
600.0 (0.0)
563.5 (9.9)
65.8 (3.8)
4.3 (0.5)
81.9 (7.1)
Uuility
391.3
388.0 (4.1)
383.7 (3.1)
343.3 (4.4)
89.4 (6.7)
348.7 (3.5)
Green
FWER
0.0%
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
Instances
Switches
0.0
0.0 (0.0)
0,0 (0.0)
0.1 (0.0)
0.9 (0.0)
0.1 (0.0)
(48.0%)
Success
99.1%
98.8% (0.4%)
97.3% (0.4%)
87.6% (0.7%)
26.6% (2.0%)
89.3% (0.3%)
T-to-S
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
605.8 (1.2)
602.2 (1.3)
T-to-F
600.0
600.0 (0.0)
710.1 (33.9)
54.8 (10.0)
(1.5)
77.2 (10.6)
Utility
263.5
-316.2 (18.2)
-13.1 (3.6)
(8.8)
14.1 (2.4)
67.6 (6.4)
Amber Instances
FWER
0.0%
0.2% (0.3%)
0.1% (0.3%)
0.1% (0.3%)
0.0% (0.0%)
0.1% (0.3%)
Switches
1.0
0.0 (0.0)
0.7 (0.0)
0.8 (0.0)
(0.0)
0.9 (0.0)
(29.9%)
Success
97.2%
28.4% (188%)
22.6% (1.1%)
14.7% (1.5%)
6.3% (0.8%)
39.3% (2.3%)
T-t-SS
600.0
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
(17.5(6.1))
670.6 (6.1)
T-o-F
600.0
(0.0)
765.5 (11.4)
(6.0)
4.3 (1.3)
126.0 (16.6)
Utility
0.0
-588.3 (4.6)
(12.2)
(4.5)
-3.8 (1.1)
-41.7 (3.0)
Red
FWER
0.0%
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
Instances (22.1%)
Switches
1.0
(0.0)
(0.0)
1.0 (0.0)
1.1 (0.0)
(0.0)
Success
0.0%
1.2% (0.5%)
1.0% (0.5%)
0.5% (0.2%)
0.2% (0.2%)
1.6% (0.4%)
T-o-S
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
600.0 (0.0)
654.7 (218)
T-to-F
0.0
600.0 (0.0)
(S.1)
39.4 (4.2)
4.2 (1.0)
46.4 (3.9)
that be conducted by each algorithm as most two, and specify a F- for algorithms that can potentially run more than one experiment- (Greedy) Bayes-OCP-whiie we still specify a (0.95) for algorithms that always run exactly one experiment- algorithms are bounded by %5. Results are given in Table Bayes-OCI still performs the best when explicit control of FWER is required.
adaptive enrichment and
RCT and futility stopping. These specifications ensure that FWER of all
C EXPERIMENTS WITH MISSPECIFIED OUTCOME DISTRIBUTIONS
We consider the same setting that we have considered during our experiments in Section 6 Except now, the ground-truth outcome distributions such that, when y ~ Ωg y with probability (0,+1), 1)/2 and y -1 otherwise. In order to ensure that 0₂ E [-1, 1]. we also sample ground-truth mean outcomes so that 0~ 2p- 1 where p distributed according to Beta distribution with a 979/200 and / = 801/200 (note that the mean and variance of 0₂ remains the same as in our original experiments). Despite the fact that outcomes are now distributed in a Gaussian way, we leave the implementation of Bayes-OCP unchanged, which still assumes that outcomes distributions are Gaussian. So, there is now a mismatch between the structure of outcome distributions specified as part of Bayes-OCP and the ground-truth outcome distributions. Results are given in Table 6 Bayes-OCk OCP still does not inflate FWER despite the misspecified outcome distributions.
D EXPERIMENTS WITH MORE STOMIC-POPULATION
We repeat our main experiments with more than two coomicpoppulttonns specifically we set | = 10. As before, all stomic-populatie
have equal propensities such that 7, = 1/10, Vz E X, and the meta-experimenter has the same positively-biat biased prior for the mean outcome associated with each atomic population: Og ~ (0.1, 0.1). Vx X. We randomly generated 100 environment (repeated five times to obtain error bars), and the results are given in Table ? We observe that Bayes-OCr still performs the best. These results confirm that greedy approximation is suitable in identifying candidate experiments when the number of atomic-populations is large.
E SENSITIVITY ANALYSIS
Bayes-Oce OCP has one parameerr 3, which controls how optimistic the switching rule given in line 14 of Algorithm i is, from B 1/2 meaning decisions are made greedily to B =1 meaning
17

---

Published as a conference paper at ICLR 2023
Table 7: Performance comparison when the number of iromiic-popllationn is 10.
Adaptive Enrichment
Futility Stopping
Greedy
Algorithms:
RCT
Bayes-OCP
w/ Bayes-OCP
Bayes-OCP
Utility
8.0 (39.2)
143.9 (31.1)
141.0 (27.5)
40.3 (5.9)
172.4 (23.8)
All
FWER
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
0.0% (0.0%)
Instances
Switches
0.0 (0.0)
0,4 (0.0)
0.5 (0.0)
1.0 (0.0)
0.6 (0.0)
(100%)
Success
60.8% (3.9%)
71.0% (3.6%)
51.2% (4.7%)
15.6% (2.6%)
63.2% (4.2%)
T-to-S
600.0 (0.0)
678.9 (8.3)
600.0 (0.0)
648.5 (13.2)
647.5 (4.4)
T-to-F
600.0 (0.0)
672.7 (58.8)
130.4 (12.5)
9.3 (5.0)
200.5 (72.2)
200
130
100
Dnilyy
50
RCT Aduptive Eurichunnn
Rayes-OCPP
-30
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
0.99
Hyper parame-arttterr
Figure 5: Utility achieved by Bayes-OCP for various values of tlpeer-prrmeeter B.
decisions are extremely optimistic that the original experiment will never be abandoned (as there will always be chance that it succeeds). As with all online algorithms, tuning 3 is challenging since no a priori data would be available perform cross validation. However, nice feature of Bayes-OCP is that / is rather interpretable, it is the evidence required against the ongoing experiment: An alternative experiment preferred over the ongoing experiment only if is believed be the better experiment with at least 3-confidence. We evaluate the sensitivity of Bayes-OCP's performance to hyper-paramett in Figure Bayes-OCP performs better than an RCT for configurations and better than adaptive enrichment for most configurations.
F FUTURE WORK
Extending the scope of Bayes-OCP One limitation of Bayes-OCP is that only adapts the target population X C X' of experiments but not the sample horizon or the success criterion p. We have chosen to focus on the selection of a target population since we believe the target population of an experiment be the most critical design dimension adjust adaptively. As we have already highlighted in our introduction, experiments with inflexible target populations can be problematic when responses the treatment of interest are highly heterogeneous.
That being said, the high-level strategy of our proposed algorithm should still be applicable to adapting design dimensions other than the target population, namely and p. At high level, Bayes-OCP first identifies a candidate experiment and then compares the identified experiment to the ongoing experiment n optimistic manner. Regardless of the given set of viable experiment design V, one could still follow the same strategy; the only complication would be to adapt how candidate experiments are identified depending on what design dimension varies across experiment designs V.
For instance, when experiment designs varied in terms of X, combinatorial search was required to identify good candidate experiments, for which we proposed a greedy strategy. When experiment designs vary in terms of p. simple search over all possible would suffice for identifying candidate experiment. The case where experiment designs vary terms of is more complex; optimal 7 for an experiment would be dependent unknown effects oz; selecting good candidate experiment would involve estimating the optimal 7 given posteriors over f,.. This would be an interesting problem to explore future research direction.
18

---

Published as a conference paper at ICLR 2023
Performance guarantees While our theoretical results motivate the general use of an optimistic decision rule, they do not provide any guarantees about the performance of the specific rule we propose as part of Bayes-OCP Another future research direction would be to prove an upper bound on the sub-optimality gap of Bayes-OCP.
G FURTHER DISCUSSION ON MAIN RESULTS
Table report six metrics: Utility, FWER, Switches, Success, T-to-S, and T-to-F. We have already discussed the implications of Utility and FWER in Section Here, we highlight other interesting phenomena regarding the remaining metrics. First, we see that Greedy Bayes-OCP switches ex- periments much more frequently compared with Bayes-OCP. This is because Greedy Bayes-OCP requires less evidence against the ongoing experiment when comparing it against an alternative experiment, whereas, Bayes-OCP favors the ongoing experiments more, Second, we see that a higher success probability does not necessarily also imply a higher utility. For instance, compare RCT with futility stopping, futility stopping is able to achieve higher utility than RCT by terminating risky experiments early and saving costs. However, this of course also means that futility stopping sees fewer experiments to completion hence leads to lower success probability. Finally, we see that succeeding failing early does not necessarily imply a higher utility either. Our best algorithm Bayes-OCP succeeds the latest on average as well as fails the latest compared with other benchmarks favoring red instances. This highlights the importance being conservative when making decisions. being optimistic, and favoring the status quo more than a potential adaptation.
H FURTHER DISCUSSION ON RELATED WORK
Multi-armed bandits The optimal commitment problem is similar to multi-armed bandit (MAB) problem (Auer et al. 2002 Bubeck et al. 2012) in some aspects: Like arms in a MAB problem, each experiment design has random utility given by R5g -D(,) TCa. where D, is the source of randomness, and the distribution of this utility is unknown. Also similar to a MAB problem, the overall goal is to sequentially select experiment designs (cf. arms) that yield the maximum cumulative utility. The main difference between the two problems is that, in a MAB problem. selecting an arm immediately reveals a sample from its random utility, while in optimal commitment, running an experiment &i just for one time step only incurs a cost of Co; observing full sample of its random utility requires the experiment to be run until its completion for consecutive time steps, without selecting any other experiment design in the meantime.
One can naively apply a MAB algorithm by viewing each viable experiment design as a unique arm, and by running experiments/arms selected by the algorithm until their completion to observe full samples from their unknown utility distributions. However, this obviously side steps the main question we want to answer in optimal commitment: When can we abandon commitment-in this case, the decision to run an experiment/arm selection until its completion- before fully observing its outcome? Looking optimal commitment from a MAB perspective reveals that there are two epplore- dilemmas present in optimal commitment: One is with respect to which experiment to select next, and the other is with respect when preemptively stop the current experiment (i.e. breaking a commitment). MAB algorithms address the former dilemma but not the latter.
Task replication in parallel computing There is work (Ghare and Leutenegger 2005 Wand et al. 2014 Wang et al. 2019) that focuses on the problem of when to kill existing tasks and relaunch them in parallel computing, which is related to optimal stopping/switching. However there, the focus is on reasoning about when a stochastic event (i.e. successful completion of a computational task) will occur without any extra information other than the fact that the event of interest has not occurred yet. In contrast, in our setting, the reason about the random outcome of an event that is scheduled to happen at a deterministic time point (here, the event is an experiment reaching its conclusion). This means that our problem has a completely different information structure when compared with the problem of task replication. More formally, we observe samples y: that are informative of whether p(D,) 1 when 7 is fixed variable. In contrast, the problem of task replication would correspond to the setting where 7 is a random variable with a known distribution and p 1 always holds (hence no need to observe any samples yt). Among optimal stopping/switching problems, the structure of our problem is more closely related to sequential hypothesis testing, which we have already covered in Section 5
needs to process a streaming set of samples to
19

---

Published as a conference paper at ICLR 2023
1 PROOFS OF PROPOSITIONS
1.1 PROOF OF PROPOSITION I
We start by relating the optimal value function V* to the optimal Q-function Q". Letting T; = T7",
VO (4,
t) = > -() - C- (min - t])
+1(7"(4,1)) = (R 1[7 >T] p(--) - (l (minn(T 1))lpe [4]
+1(r"(t,pb) = (R 1[T > > p(++) C (min(77+1,77 -t)]lu p]
(11)
= (oo) E[R 1(771 (.}pp())) - C. (min - t) 14]
(12)
= -] (t,,,
(13)
(t.,))
(14)
where (II) holds since T (t) =0 =
T -7- and x²' v)
Tr min 22: (t', (1)) = 2
x" ((+ Vo} when conditioned on μr μu, and [3]
112 holds since
M4 1 1f 1f="(t, (t. Vo) and Tū 1
holds since xt pt. = Vn = (t( > Intuitively, the maximum possible value at a given time is achieved either by stopping immediately or by conducting the experiment for at least one more time step and then following the optimal policy thereafter.
Next, we observe that
P(p++1 $ 4e |m = n} PPPeet1 S 11100
p)dP(0)m mr}
/ F n1' 0 41
1
1(0 1o: 1/1)d0
t+1
11+ 1)2
// 1 S 107
0 1p
[M+11
5(0
pr; 1/e)dp441d8
++1 (t+1)²
//
y (t + 1)le.
Mpm- S 1'} 1441
11+ f(u: 1/1)dpe
1+1
y + (1 + 1)1s
1l M(05 ((4;)))))))))) 1/1)dad
1
t+1
[x+ Y
l+1 S }
Fx Y-N(0,11
X y
PXAV/CC-D- = F(y'-1;
t+ 1 S11 - 11
45 1/41)1
(15)
where f(x; = (1/ -
and F(2;0²) = (1/V 1 (11/2)) fo² de' are
the f.d.f. and the c.d.f. of the Gaussian distribution with mean zero and variance 0² respectively. Hence dP = μ v/| = /1] = f(y' pr 1/e 1/t+1)dy'.
Then, using the relationship between V and Q" and the observation regariigi P(u1+1 S pllme p). we drive the following Bellman optimality condition:
(t. ElA- 1 > >T}- p(u+)- - CC. (min(TTt): - t)]lp = 14]
- + E (Ti >T] p((=)) - C. (min (Tin - -1)114 = 1r]
--C + f 1(T > r} p(u-) C. C. (minnfTt::: - -1)lm4+1 = =A²ll
ddP = -1'm== =4)
=-C+/V*(1 + 1,1 =1/114 = 11) =-C+JV +1,4)) 1/++1)d1
20

---

Published as a conference paper at ICLR 2023
- ( -C + f max (0, Q* [ |, 2, -//t/////////////
(16)
(17)
For the problem setting where C = 1, R - 2, 0. and r = 2, we have
(1, max (0, -0 -V= + -)[(=;1/2)dz
max
/
max 0, 0,-1+
f(z; 1/2)dz
max(0,-111
2F (p: 1/2)}
if < 0
-1 11 (F2 if 2 0.
Notice that, for μ > 0,
dc
Va (1, = de (1+2F(uu;/))
d1p
dn2
d
(25(er 1/2)
dp
-4/xx)) x²² <0
hence V*(1, μ1) is concave at least on interval μ E (0, oo) and is not a convex function. More- over, -V* (1, cannot be convex function--o equivalently p), cannot be a purely con- cave function- eitherr For an arbitrary E (0). V (1, u) > 0 and (1.1-pp) = 0 hence (1/2)V*(L,, (L.
but (1.22)
= (1, =
1.2 PROOF OF PROPOSITION
We will prove the proposition by showing that
(i) (t) is non-decreasing in that is < n² = (tt.sQQu (Q
lim (* n)) (7 +)C R > 0, and
(iii) lim----0 Q" (1, <
for E (1,...., 1} via mathematical induction. Notice that these three facts together with the
fact that Q' (4) is a continuous function in for & E (1,...,, -
ww-would imply the existence of
a unique ff such that Q*(tt, m4) = 0, Q (t. > 0 + 4 > ue, and (t, u) S 0 = μ S 14, which in turn would imply that
Lo Ø
if mi (t,t,)) >0
(t.
if S pi =
(t, 0
meaning the optimal policy x² is indeed of "thresholding -type" as the proposition states. First, we observe the following base cases for t 1:
(i) Q- (T - 1, is non- decreasing in μ. When μ < n²,
Q" - 1, = -C +
+ 211(::=/1-1=
(18)
= - - - 1/-)dz
< CCC+Rf1fu +2> /Tt///;((;;;;//11) - -1/r)dz
(19)
= Q"((-L1,p')
where 18) due 16 and 19 holds since ,+ / = j'+2> 1+2 V/VF.
(ii) 1om (* - 1,14 = - C + R > 0 since
lim 0"(7-1,11) 1, #+000
lim ( (- C // 1/7)dz)
#+000
21

---

Published as a conference paper at ICLR 2023
lim
-C+R f(2; - 1/r)dz
10000
Ja/VT-a
C + R2f(2; 1/-)dz
-C R.
(iii lim 1, = -C < since
lim
Q" - 1, 10
lim ( (-C C / -1/-)da)
17000
-C+R 0
lim
(1z)\(r-1)-1//)))
11-0
Ja//T-R
jalvi-u
lim
CCCR 1
((zi 1/5)dz) ))
11-00
-C R(1 -
=-C
Then, we show that the following inductive cases hold for / E - -1....,
{r-
(i Given that (t, ju) is non-decreasing in μr, (t 1, 11) is also non-decreasing in μ.. Similar to the base case, when < Jd,,
Q' - + (, (tttttt))) ((11)))) -1/)ddz
(20)
S |/aax(00 (t, (2) - 1/1)dz
(t-
where 20 is due
17
and also given that Qt, (t. (J) is non-decreasing (ii) Given l Q* - () R-and
in we have l Q* - 11) - 1)C R 0, which can be shown using the sandwich theorem:
Qt -C' + ma
f(, 1/0-1) 1/0/dz
< -C + max (0. lim (t,,)))){((,, (/t-11111/1)----)) 1/ejdz
f -C + (-(r - -tCC+ f() 1/(--1/-111 -1/i)dz
-(r - + R.
(21)
Q" - -C ma Q' (t. H + +
2 CC+ [
| ma (t, =)) 1/))dz
/
2 -C+
ma (* (t, |n| 1/C-1) 1/1)dz
2 - 1w( /
1/2,1/(011- -1/)dd2,
(22)
Finally, observing that
lim 22
lim
-
+0000
+1000
C (lm Q (t,,)')) 1/1-1) - 1/2)dz
= =- - 1)C R,
together with bounds (1 and (2). we obtain lim Q* 1, 11) = 1)C
(iii) Given lim oo Q" (t. and lim Q" (t, 4)
C < 0 also given that (t, 11) is non-decreasing in
that μ4 exists- we have lim Q* (t 1, =
which again can be shown using the sandwich theorem:
Q' 1, -C + max(0, (1 22) 1/1-11 1/t)dz
22

---

Published as a conference paper at ICLR 2023
2 CC
(23)
Q- 111 = -0+ + 10 1/0ddz
-f50 Q' 2)5,,,=))((,,/// 1/1--11111)--) 1/)ddz
(24)
HI-H
S S R
f(z, (1-1) - -/))dd
(25)
S S + R - QUi-"
- [(z, 1/11-11 -
(26)
where 24 holds since (*, μ +2)
2)> > 0 and only if > 14 - H and max (0, (t. =)} 0 otherwise, and (25) holds since (1, n)) < dim - for all μ as Q" (1, 1)) is non-decreasing in j.. Finally, observing
Pi-H (2(111111 1- 11
[(z 1(1-1) 1-/(t-11-1///p00 1/j)dz) ))
lim
26
lim 1--
- C 18
=-C,
together with bounds 23 and
26). obtain lim Q' (1) = --C.
When put together, the base cases and the inductive cases above imply that conditions (i-iii) hold for all t {l...., 1} hence 147 exists for all {1...., 1} which concludes our proof. D
1.3 PROOF OF PROPOSITION
First, we prove the existence of jiroodyy for all t E TO...., 1} by driving an analytical formula for ( n) = va (0) (1, Letting T((0) T7
(t, = E[R 1(T > p((r) ( -C (m) ,7] - -t)lur =
EE( p(u,) C- (r t) =
(27)
= -C + [E p(1+) - CC.
(r (r-t pr-t- -- --1))ett+= = 1JJap(u44 =m'mm 1a)
--C+ 1,
=n'lp-= 11m==11 #))
-CC+
- 121/:-1//444)d/
(28)
-C+ -C (f:11++)))):::/=//-00=)
(29)
where (27 holds since n. (0)) p)) Vo for all and j hence always the case that T:()) 0o, and 28 due to 15).
In the remainder of our proofs, we take a = 0 for notational brevity. This is without any loss of generality as, by simply shifting each value function and Q-function by a/ /F with respect p. all of the following arguments would still hold. For a = 0, we show that
v = -) t)C R.F
(30)
1/-2 1/7-1/7
for all E (1....,,T- 1} via mathematical induction. Note that 30 is true for 1:
((0 (- -C' -
1/7)dz
= --C --)=://(()---)- - 1/-)dz
- R
f(2;1/(-11)-1/ 1
=-C+ R
f(2:1//-11--1/1 1/7)dz
M/r-
= =-C+
f(2;1)d2
-000
23

---

Published as a conference paper at ICLR 2023
p
= R.
Vir--1)-11
where F(x)= = F(x; 1) is the c.d.f. of the standard Gaussian distribution. Moreover, assuming 30 true for t. it is also true for t -
v (t (t- - 11)
= JV- (t 1/0)dz
= -- -1- 1)C + V/: 1/2)dz
-(r-++1)C
+ (uta)//11
= --(r - 1+C
+ R /I p+t
=-(7 - 1)C
+ R +R //
112 <u4aafe2t/////err///-- 4/))) dz
-( - + 1)C R. P... (Z' 11+ [
Z' ((1(0//1-116
2'-Z
- - 1)C+ P.
z-Z N(0.1) 11
1/(1-1)- -1/- S
V/((-1)-
11
= - + F
1/(a-1)-11
Therefore, 30 indeed holds for all ( 1}.
Next, we observe that (1, (4) has root at μ = F- ()C/)))/////// provided that (T - (r-C)C/R E (0, 1). which is the case for all t {l,..., since < Morcover, V V (t, is a strictly increasing function in Hence, there exists a unique 14 gready for all t {I,..., 1} such that ((() (t., 14t (t., ! > 0 and (10) 1)) 0 > 1 pt greoyy In other words, qrreedy is also thresholding taypoldiigngegeeeeeoe policy as the proposition states.
p1)
Finally, we have (t) (4) (()) 14) & (t, m4] = 0 hence jot S 1u greedy This is because, by definition, (1, 2 Q (1) for all L, μ for any given policy æ, including (7))
1.4 PROOF OF PROPOSITION
As in the proof of Proposition we take a = 0 for notational brevity. Once again, this is without any loss of generality as, by simply shifting each value function and Q-function by a/v? with respect to all of the following arguments would still hold. Remember that the formula we derived for (0) (t. (J) in 330 holds when a = 0.
We start by deriving two bounds on the optimal Q-function Q* p): (i) lower bound and (ii) an upper bound. For the lower bound, it is sufficient to observe that
v (t. (4) = Q (t, S Q" (t)
which holds since, by definition, (t, p) 2 Q*, μ1) for all t, for any given policy π.
For the upper bound, we use mathematical induction to show that Q" (t, & ( First, for the base case of
Q- 1,11 -C -C+++
(31)
=-C+ [++filut>>>](<:>// 1/c+1)dd
24

---

Published as a conference paper at ICLR 2023
- [V 2)f(2; 1/-1))d2
(T 1.11.
(32)
where (31) is due to [6]. and (32 is due to (29) Then, for the inductive case, assuming Qt,
(t,, μ))
1 + ( (t, p).
(t 1,1) -C max (*, 2){ 1/0-1) f/
1/2)2z
(33)
(0 ()),, (2, - 1/1)dz
(34)
( 1)C + [V' (1. =)ʃ(z, 1/1-1) 1/1ddz
= - (1
where (33) is due to 17 and holds since -C Q" (t, implies that max(0, Q" (t, -
max (C (t, p)) Q (t. p)) CC+ (t.
Define ne and 24 such that
(1, =0 = 1t = F-1
(T-
Vi
R
C R
(T (T- (t.44 (+) =0
= + = F-1
T
which we are able write in closed form using the formula we derived for V (t., p)) in 330 during the proof of Proposition
By definition, H4 greedy =Mi Morcover, (i) ((0) (4) Qt, (z) = = (t) pt due to
lower bound, hence Mi S 1h] (remember that ((0) 1) strictly increasing function in n), and (ii) (T (T- 1-C + (0) (1, 14, 00- Q' (L) S (r - -- 1)C (L))(, due to our upper bound, hence (00 u, S ( ( pt) meaning Pt μt. Putting together these facts, and also the fact that μ S preedy we obtain int Pt pready < M 14 the proposition states.
J BENCHMARKING ALGORITHMS
Algorithm 2 Adaptive Enrichment, Futility Stopping with Bayes-OCP, Greedy Bayes-OCP
I: Initialize Ns and a for all X
2: X + +-,, t Do
Start experiment sb P.)
4: loop:
5: 1-
6:
Observe Ft, V1
D. D (2r, 31}
8:
1/02, + 1/002, +1
9:
Mr, f Mr. + (y+ 10
10:
X t+
11:
while X X
I' arrgaax, ex
or By N (X'U (0,))]
13:
if Eg -i 9 (x' U u (0x)) > Bo g g (x':
X + C {a*}
14:
15:
else:
16:
break if Adaptive Enrichment
17:
and t 7/2 and Egg-Nire (9 (0 (X': [0+1)] +g) Dr; (0z))):
18:
X - X', + O, Do < Ø Start new experiment 1/ (X, if Greedy OCyes-OCP and Eg N
19:
20:
(0) (X': Ea, ,3 (0.1)]:
21:
X +- X'. t- 0. Do 2 Start a new experiment if Futility Stopping with Bayes-OCP and Po (o (0. ((x)))((x)>))00
22:
(X, T.
23:
0(X, ( 8:
24:
Stop all experimentation
25

---

Published as a conference paper at ICLR 2023
K
GLOSSARY OF TERMS AND NOTATION
Term
Notation
Description
Experiment
Conducted to confirm efficacy of an intervention, e-g. new treatment in clinical trials, or new recommendation policy in online advertisement
Subject
Individual participant of experiment, c-g. patients in a clini- cal trial, customers in online advertisement
Population
C x
Collection of subjects that share the same qualities, e.g. female patients in a clinical trial, or all customers with the same preferences in online advertisement Indivisible populations
Atomic-ponn populiatoon x
Propensities
n
The probability that subject being from atomic- nomic-population xr The probability that subject being from population
yx
The probability that a subject being from Mlmmic-populatie ir conditioncd on the fact that they from population X
Outcome distribution
Distribution of outcomes that is indicative of the effect of the intervention of interest for soomic-ooullation
Mean outcomes
0,
Expected ouicome, i.e. effect of intervention interest, for atomic-population a
0x
Expected outcome for population X
Experiment design
(X, T.
Target population X, sample horizon and success criterion p
that characterize an experiment
Viable experiment designs
V
Experiment designs that can potentially be followed by a meta- experimenter
Meta-experimenter
The decision-makien agent that decides when run experi- ments according to which experiment design in W
Sample/time horizon
An experiment terminated when =
Success criterion
An experiment is declared a success if p(Dr) = Data collected by an ongaing experiment at time step D.
P
Online dataset
D
Data collected by the i-th experiment run by the meta-experi- menter at time step
Aggregate dataset
D
Collective data collected by all experiments to time step of the i-th experiment
T"
Number of time steps for which the i-th experiment con- ducted until was stopped or its time horizon was reached
Cost
Ce
Cost incurred time step by running experiment Reward received if experiment is successful Sum of costs and rewards received after all experimentation is concluded
Reward
R
Utility
G
Policy
becision-mmaiing policy of the meta-experiment
Optimal policy
n²
The optimal policy that maximizes utility G in expectation
Greedy policy
Bredy
See Scction
Test statistic
Pe
the simplified case Section the empirical mean outcome
Value function
v" (L)
The expected utility of following policy л when μı:
Q-function
Q (t.
The expected utility of following policy after conducting the ongoing experiment for one more time step when μı: /
Tr
The first time step at after time step that policy decides to stop all experimentation
Optimal value function
V
The value function associated with "7
Optimal Q-function
Q'
The Q- D-function associated with x=
Thresholds
Decision Deeision-making thresholds associated with n²
Decision-making thresholds associated with Breddy
26

---

Published as a conference paper at ICLR 2023
Term
Notation
Description
Conditional power function
Px; (8.
The probability of hypothesis test being successful condi- tioned mean outcomes {0..
Expected utility function
0(x
The expected utility of fully committing to an experiment and
waiting until it terminates when the experiment targets popula-
tion X, is currently time step 4, and collected dataset D,
Posteriors
N(H2,
Posterior distributions over mean outcomes (0x) maintained by Bayes-OCT CePes-OCPPa such that 0₂ (D ~ N ~Nuu o
27


Reviews:
[{"review_id": "vA6VrM4Zds", "reviewer": "Reviewer_tD32", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the setting of optimally breaking commitments. In particular, it considers a decision making problem in which a decision maker has to decide on whether to break a commitment and if so, whether to select an alternative commitment or stop the making commitments. The paper formally characterizes a simplified version of the problem, which leads to three key insights that are later on used to design a novel algorithm for a more general setting. This novel algorithm is validated in a simulation-based test-bed. ", "summary_of_the_review": "While I believe that this paper studies an interesting topic, and provides an interesting set of initial results, the current results look preliminary at the first glance and could be further extended. More concretely, the theoretical analysis focuses on a simpler instance of the problems setting, while the experiments are based on simple benchmarks. In my opinion, the paper could greatly benefit from adding experiments that tests if the proposed algorithm is brittle to the deviation from the distributional assumption it relies on, and potentially adding more sophisticated baselines. ", "strength_and_weaknesses": "**Strengths**:\n- Arguably, the main strength of this paper is the novelty of the paper is the problem setting. To my knowledge, this particular setting has not been studies in the literature.  \n- The characterization results on a simpler instance of the problem setting are intuitive, but provide insights that one can use to design effective algorithms. These results also inspired the design of the propose algorithm for optimally deciding on commitments. \n---\n**Weaknesses**: \n- The more complex, and presumably more interesting setting is not formally analyzed. This makes the set of characterization results somewhat incomplete. \n- The algorithm assumes that the outcomes of interest come from a specific distribution, which does not affect the soundness of the algorithm, but may affect its performance. \n- The proposed algorithm doesn't seem to have any provable guarantee that it converges to an optimal solution, and it is not clear how it would fare against e.g. policies learned via RL with function approximation. The paper only reported results with relatively simple RL baseline.  \n- Benchmarks appear to be rather simple with only two atomic population and carefully chosen parameters. Although the experimental results indicate that the proposed approach is more effective than the baselines, more extensive experimentation would be quite useful. For example, the experimental setup where the distributional assumption on the outcome could fail to hold would be informative about the robustness of the proposed approach. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pZD6YNloCo", "reviewer": "Reviewer_1tQ1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of online decision making, in which the decision maker decides whether to commit to long-term actions until success or failure, or stop it and/or switch to an alternative. The authors formulate it as an optimal commitment problem (OCP), which is a new type of optimal stopping/switching problem. For a simplified case of OCP with a single commitment, the authors provide theoretical analysis to obtain the characteristics of the optimal solution, and then propose a practical algorithm called Bayes-OCP to solve the general problems. Numerical experiments are also provided to validate the superiority of the proposed framework compared to related existing methods in the literature. ", "summary_of_the_review": "Overall, I find the paper is generally well-written and contains some good results both in theory and practice. But there are also several points for improvements as detailed above.", "strength_and_weaknesses": "Strengths:\n* The theoretical analysis of the simplified OCP with a single commitment is insightful, which not only motivates the general algorithm but also provides a better understanding of the OCP framework.\n* The numerical experiments are relatively thorough and the discussion about green, amber and red instances are also inspiring. \n\nWeaknesses:\n* The proposed algorithm, Bayes-OCP can only handle experiment designs with fixed time horizon $\\tau$ and fixed success criterion $\\rho$. The only decision variable becomes the targeting subpopulation $X\\in\\mathcal{X}$. This limits the applicability and practicality.\n* The numerical experiments are conducted only on a synthetic/simulation environment with only two candidate targeting atomic populations. It would be more convincing if some experiments on more candidate targeting atomic populations and real-world datasets are included.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gsiFdmGt-k", "reviewer": "Reviewer_jmHo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors formulate the optimal commitment problems (OCP), which can be used to model problems in which the agent determines whether to continue with some long-term actions like experiments until a signal of success or failure is received or to stop it in the middle. The agent is also allowed to change to a different (long-term) action during the process. On the theoretical side, the authors characterizes some fundamental properties of the optimal solution in a simplified OCP problem that has a single commitment. On the empirical side, the authors then propose an algorithm, Bayes-OCP that can be used to solve more general OCP problems. The authors also conduct  experiments to demonstrate the advantage of the proposed model and algorithm over existing models/algorithms in the literature.", "summary_of_the_review": "Overall I think the paper contains non-trivial contribution in terms of both modeling, theory and practice. It's also clearly written and the reproducibility is also good given the detailed experiments. But the authors should also take a further look at the potential weaknesses detailed above and try to clarify and/or fix them. ", "strength_and_weaknesses": "Strengths:\n* The framework is well motivated and the applicability of the model to different domains like healthcare, finance and energy is clearly stated and demonstrated.\n* The theoretical results of the special case OCP solution characterization are inspiring. The numerical results are also generally convincing and supportive. \n* The appendix also contains many interesting discussions, including the POMDP formulation of OCP, RL benchmark, the Bonferroni correction to improve FWER and the robustness to outcome model mis-specifications.\n\nWeaknesses:\n* If I understand correctly, it is assumed that the $R_{\\psi}$ and $C_{\\psi}$ coefficients in the utility function are known a priori for each experiment design $\\psi$, which again limits the applicability and practicality of the proposed framework. Also some explanations are needed to characterize when such an assumption holds and when not.\n* The figures (Figures 3 and 4) in the numerical experiments are not very easy to follow (the tables are good and clear, though). Some more explanations on them (especially how the switching points are determined, and how to associate each algorithm to each of the curves, etc.) would be very helpful.\n* The discretized benchmark for solving POMDP formulation is not a very fair comparison. There are lots of more efficient POMDP algorithms (both with and without fully known models) that can be used (and many off-the-shelf solvers are available such as https://github.com/JuliaPOMDP/POMDPs.jl), and the authors should better compare with them in order to argue that the POMDP + RL approach does not work (or not). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aWpr36ueim5", "reviewer": "Reviewer_jpJi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the _optimal commitment problem_, where an agent must decide, at each time step $t$, whether to continue running the current experiment (i.e., _commit_ to the current experiment), or terminate the current experiment, switching to a more promising experiment (or to no experiment at all, if one does not exist). The paper first analyzes the simple case, where the decision is just whether to continue or stop the current experiment. The paper establishes that the optimal policy, in this case, is a threshold policy (i.e., if the outcome of the current experiment is above some threshold continue, otherwise stop) that becomes increasingly close to a greedy policy as the experiment approaches its end. \n\nThe paper then proposes Bayes-OCP, an algorithm that maintains a posterior distribution over the mean outcome for each possible atomic population $x\\in\\mathcal{X}$. Then, at each iteration, assesses whether switching from the current population $X\\subset\\mathcal{X}$ to an alternative population $X’\\subset\\mathcal{X}$ (including the “empty” population) is likely to lead to a higher utility. In that case, the agent stops the current experiment and starts a new experiment involving the population $X’$.\n\nThe results in the paper show that, when compared with alternative approaches, Bayes-OCP strikes a good balance between optimism (i.e., not stopping the experiments too soon) and timeliness (not stopping the experiments too late).", "summary_of_the_review": "The paper is, in my opinion, very well written and provides a novel and interesting contribution. I would like to understand better the applicability of multi-armed bandit problems to the setting considered in the paper.", "strength_and_weaknesses": "The paper is somewhat outside my comfort area, so it is possible that I misjudge some of the contributions of the paper.\n\n**Strengths:**\n\nThe paper is very well written, and a pleasure to read. It makes an excellent work of motivating and setting up the problem, the contributions are clearly exposed and presented in a constructive and intuitive manner. The experiments, although involving mostly simulation, do support the claims of the paper adequately. \n\n**Weaknesses:**\n\nI have, perhaps, two main complaints regarding the paper. The first is that I would have liked to grasp more clearly the relation between the proposed work and other decision-theoretic problems. The paper properly provides the connection between the proposed framework and POMDPs, and also establishes early in the discussion that the OCP problem is, at its essence, one of exploration vs exploitation. This said, it would seem to me that approaches based on sequential prediction/multi-armed bandits (particularly using experts) could also be relevant alternatives to consider, but I found little discussion on this problem. Since $\\mathcal{X}$ is finite, an approach that seeks to identify which $x\\in\\mathcal{X}$ to select that maximizes the utility would be a relevant competitor. I’m probably missing something here, but some discussion on this would certainly help to clarify my possible confusion.\n\nMy second complaint is more on form than on content. Personally, I found that laying out the tables and figures in the middle of the text to be disruptive while reading the paper. I realize that this is probably done for a matter of space, but perhaps moving some of the tables/plots to the appendix may help (for example, Figs. 3 and 4 can probably make it to the appendix with no significant loss in the exposition).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XHDub6kpOyf", "reviewer": "Reviewer_mtcX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a problem setup in which decision makers launch experiments (say over target population of agents) and have to wait a period of time before receiving the outcomes. These outcomes may or may not be conclusive, and accordingly the experiment launched may or may not be useful. If found inconclusive, it may be useful to relaunch the experiment with a new sample population. This paper studies the problem of determining when it is optimal to continue or break the commitment to a particular instance of an experiment. \n\nThe paper formulates this as an “Optimal COmmitment Problem (OCP)” and provide some theoretical analysis. The paper also proposes a practical algorithm for solving the OCP problem and evaluates it empirically in the context of clinical trials. \n", "summary_of_the_review": "New problem domain, well-motivated, first results in terms of theory and algorithm.\n\nGood paper overall", "strength_and_weaknesses": "Strengths\n\n— Well motivated: The paper describes application of this problem formulation to multiple domains and motivates the applicability of these techniques well.\n\n– Well written: The paper is well written overall and is easy to follow. Inclusion of warm-up section for example is useful in building understanding. \n\n– New formulation: The OCP problem formulation seems novel to the best of my knowledge.\n\n– Good results: The proposed methodology and algorithm is shown to perform well.\n\nWeaknesses\n\n– Theoretical guarantees on algorithm performance would be valuable\n\n– Related work: There is some related work on queuing theory and optimization in cloud computing (completely different application domain) such as “Efficient Straggler Replication in Large-scale Parallel Computing” by Wang et al. which focuses on the problem of when to kill existing tasks and relaunch. It may be interesting to discuss if these techniques are relevant or if there are fundamental shortcomings that prohibit their use for the OCP problem.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster