Should I Stop or Should I Go: Early Stopping with Heterogeneous Populations
Authors: Hammaad Adam
Fan Yin
Mary Hu
Neil Tenenholtz
Lorin Crawford
Lester Mackey
Allison Koenecke
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Randomized experiments
heterogeneous effects
causal machine learning
fairness
sequential testing
clinical trials
A/B testing

Abstract: Randomized experiments often need to be stopped prematurely due to the treatment having an unintended harmful effect. Existing methods that determine when to stop an experiment early are typically applied to the data in aggregate and do not account for treatment effect heterogeneity. In this paper, we study the early stopping of experiments for harm on heterogeneous populations. We first establish that current methods often fail to stop experiments when the treatment harms a minority group of participants. We then use causal machine learning to develop CLASH, the first broadly-applicable method for heterogeneous early stopping. We demonstrate CLASH's performance on simulated and real data and show that it yields effective early stopping for both clinical trials and A/B tests.

Should I Stop or Should I Go: Early Stopping with Heterogeneous Populations
Hammaad Adam"
Fan Yin
Huibin (Mary) Hu Microsoft Corporation
Massachusetts Institute of Technology
Microsoft Corporation
Neil Tenenholtz Microsoft Research
Lorin Crawford Microsoft Research
Lester Mackey Microsoft Research
Allison Koenecke Cornell University
Abstract
Randomized experiments often need to be stopped prematurely due to the treatment having unintended harmful effect. Existing methods that determine when stop an experiment early are typically applied to the data in aggregate and do not account for treatment effect heterogeneity. In this paper, study the early stopping of experiments for harm on heterogeneous populations. We first establish that current methods often fail stop experiments when the treatment harms a minority group of participants. We then use causal machine learning to develop CLASH, the first applicale method for heterogeneous early stopping. We demonstrate CLASH's performance on simulated and real data and show that it yields effective early stopping for both clinical trials and A/B tests.
Introduction
Randomized experiments are the gold-standard method of determining causal effects, whether in clinical trials to evaluate medical treatments or in A/B tests to evaluate online product offerings. The sample size and duration such experiments are typically specified in advance. However, there are often strong ethical and financial reasons stop an experiment before its scheduled end, especially the treatment shows early evidence of harm [7]. For example, if early data from a clinical trial demonstrates that the treatment increases the rate of serious side effects, it may be necessary to stop trial to protect experimental participants [9].
A variety of statistical methods can be used to determine when to stop an experiment for harm [22, 39, 14, 13]. Investigators in both clinical trials and A/B tests will often choose use subset of these methods-collectinng Stopping tests not only identify harmful effects from early data, but also limit the probability of stopping early when the treatment is not harmful. However, stopping tests are typically applied to the data in aggregate (i.e., "homogeneousll") and do not account for heterogeneous populations. For example, a drug may be safe for younger patients but harm patients over the age of 65. While growing body of literature has studied how to infer such heterogeneous effects [see, e.g., 41J, little prior research has described how to adapt stopping tests to respond to heterogeneity.
referred to "stopping tests"-basee" on the specifics of their domain.
We continue the above example illustrate why stopping tests should account for heterogeneity. Consider a clinical trial for drug such as warfarin, which has no harmful effect on the majority of the population but increases the rate of adverse effects in elderly patients [32]. Using a simple simulation, we demonstrate that if elderly patients comprise 10% of the trial population, then applying a stopping
*Corresponding author (hadam @ mit.cdu). CCurent affiliation: Amazon. Work performed while at Microsoft (prior to joining Amazon).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Probabiltty of Stopping
Size of Minority Goup
0.50
50%
40%
30%
20%
0.25
10%
0.00
066
Harmful Treatment Effect Minority Group
Figure I: Stopping tests applied homogeneously may not stop trials in which the treatment harms only minority group of participants. For example, if the treatment harms 10% of all participants, the trial will stop less than 20% of the time, even if the harmful treatment effect is strong (boxed in red). We plot the stopping probability of commonly used stopping test (O Briien-Fleming) in situations where the treatment harms the minority group but has no effect on the majority group.
test homogeneously would rarely stop the trial for harm (Fig. 1). In most cases, the trial continues to recruit elderly patients until scheduled end, many of whom will be harmed by their participation. This outcome violates the bioethical principle of non-maleficence [37] and is clearly undesirable.
In this paper, we consider early stopping for harm with heterogeneous populations. We first formalize the problem of heterogeneous stopping and establish the shortcomings of the prevailing homogeneous approach. We then present our novel method: Causal Latent Analysis for Stopping Heterogeneously (CLASH). CLASH uses causal machine learning to infer the probability that a participant is harmed by the treatment, then adapts existing stopping tests to better detect this harm. CLASH allows an investigator to use their stopping test of choice: it is thus flexible and easy-to-use. We theoretically establish that, for sufficiently large samples, CLASH is more likely stop a trial than the homoge- neous approach if the treatment harms only subset of trial participants. CLASH also does not stop trials unless a group is harmed; thus increases statistical power while limiting Type error rate. We demonstrate these properties in an extensive series of experiments on synthetic and real data, and establish that CLASH leads to effective heterogeneous stopping across a wide range of domains.
While a small number of papers have attempted to address the problem of heterogeneous early stopping, they have relied on restrictive assumptions, limiting their applicability. Thall and Wathen [35] require knowledge of the source of heterogeneity, which is rarely available in practice. Yu et al. [42] allow for unknown groups but only model linear heterogeneity. Yu et al. [43] relax the linearity restriction but do not handle time-to Lime-to-event data commonly observed in clinical trials. In contrast, CLASH does not require prior knowledge, makes no parametric assumptions, and works with any data distribution. It is thus the first broadly applicable tool for early stopping with any number of heterogeneous groups.
We emphasize that early stopping is a nuanced decision. For example, if a treatment harms only subset of participants, may desirable to stop the experiment only on the harmed group but continue it on the rest of the population. In other situations, it may make sense to stop the trial altogether. The decision to stop a trial to protect one group may disadvantage another group that would have benefited from the treatment; any decision on early stopping must thus carefully weigh the treatment's 's potential benefit, the size of the harmed group, the nature of the harm, and other ethical considerations. Our method is not intended to replace such discussion. It can, however, serve as an aid for investigators to make difficult decisions on early stopping.
2 Background and Setup
2.1 Randomized Experiment
Consider an experiment that evaluates the effect of a binary treatment D E {0, 1} on an observed outcome Y. is assigned uniformly at random to participants the experiment. specifically to measure harm and may be different from the experiment's primary outcome of interest. For example, in a clinical trial, Y could convey a primary outcome such as mortality or a secondary outcome such as the occurrence of serious side effects. We allow for both scenarios, experiments
is chosen

---

often care about harm on multiple dimensions, Assume that an increase in Y is harmful. The harm caused by the treatment can be measured by the average treatment effect (ATE) of D on Y. ATE = F D 1| - E = 0j. A positive ATE implies that the treatment is harmful, while negative ATE implies that it is beneficial. Note that the ATE has a natural estimator: the difference in the observed means of Y between the treated and untreated groups.
Now, not necessary that every individual in the population responds to the treatment in the same way. The treatment may harm some individuals, benefit some, and have no effect on others. Assume that the population can be divided into M discrete groups, where random variable G conveys group membership. For group g. we define the conditional average treatment effect (CATE) T(g) = E D L,G g] E| 0, = g]. Note that in general, we do not observe G. Rather, we typically observe set of covariates X that correlate with group membership. We define the CATE for a participant with covariates x as (xx) = (=g1(((()))
The experiment can recruit a maximum of N participants and is run for T time steps, where the values of N and 7 are set at the experiment's 's start. For simplicity, assume that we recruit two participants at each time step-one in treatment and one in control-and so T = N/2.. For each participant i, we measure treatment assignment d; covariates Tú, and observed outcome yi, but do not observe group membership gs- The expected effect of the treatment on participant i is given by T(g)). We assume that (d1, 11, V1.91), 91), oi)...... (dy, IN. UN 9N) ) are independent and identically distributed.
2.2 Early Stopping for Harm
If the treatment harmful, it is often important to end the experiment as early as possible, to minimize harm for participants and financial loss for the experimenter. Clinical trials and A/B tests usually build in interim checkpoints, where the data collected thus far is evaluated to make a decision on whether to continue the experiment. However, it is also important to not stop the experiment at these checkpoints unless the treatment is actually harmful; incorrectly stopping early could result lost innovation or societal benefit. Achieving this balance requires using dedicated statistical methods for early stopping, collectively known as "stopping tests."
A large body of literature has studied stopping tests. Frequentist methods like the Pocock adjustment [23], O' Briien-Fleming (OF) adjustment [22], and alpha-spending [10] are common in clinical trials with a pre-specified schedule of checkpoints. True sequential methods that allow for continuous monitoring are often based around Wald's Sequential Probability Ratio Test (SPRT) [39], such as the 2-SPRT [19], MaXSPRT [16], and SiXT (mSPRT) [26, 14]. Other sequential methods are based on martingales [3, 25] and testing by betting [40]. Such tests are more practical in A/B testing scenarios, where investigators can monitor responses they arrive. There is also growing interest in Bayesian designs and stopping tests, both for clinical trials [4] and A/B tests [33]. App. A
summarizes some commonly-usee stopping tests.
While these stopping tests differ their details, many have the same general form: compute test statistic from the hitherto observed data Ån (Vim, d,:n), then stop the experiment if this test statistic exceeds some bound b(a:). Here, n is the number of outcomes observed thus far and a the desired bound on the rate of unnecessary stopping (i.e., stopping when the treatment is not harmful). For clarity, we provide a concrete example of a stopping test: the OF-adjusted z-test. Assume that Y ~ N(Du, o²), where o² is known but μ is not. The treatment effect is homogeneous, and so the ATE (i.e.. μ1) conveys if the treatment is harmful. To test between a null hypothesis of no harmful effect (Hg ATE < 0) and an alternate hypothesis of a harmful effect (H₃ ATE > 0), we use
O
1s y:d;
2E(1
V20
n/2
n/2
The OF-adjusted z-test compares this test statistic to an appropriate bound OOF (ax), which depends both the number and timing of checkpoints. For example, if were conduct one checkpoint halfway through the trial, then 1, (0.05) = 2.37. If OF N/2 > joF, the test stops the experiment for
harm; else, the experiment continues. Using the OF adjustment guarantees that if the treatment is not harmful (i.e., ATE < 0), then the probability of stopping the experiment is at most 5%.
2.3 Stopping with Heterogeneous Populations
Stopping tests are typically applied to the data in aggregate and define harm in terms of the ATE. While this approach is reasonable if the treatment effect is homogeneous, there is no guarantee that
3

---

will stop the experiment if the treatment only harms a subset of participants. For example, consider a situation in which there are two equally sized groups with equal but opposite treatment effects, that is, p(G = 0) = p(G = 1) and r()) -1).. The ATE with this mixture distribution is zero, and so any stopping test with Hn ATE v is designed to continue to completion at least (1 a of the time. Unfortunately, the failure to stop means that half of the trial participants will be harmed.
Experiments with heterogeneous populations thus require a more precise definition of harm. To comply with the bioethical principle of non-maleficence, experiments should be stopped not only if the treatment harmful in aggregate (i.e., ATE > 0), but also if the treatment harms any group of participants (i.e., 3 s.t. T(g)= > 0). We define the null hypothesis of no group harm,
Ho : [()) S Vg E (L)...M..
(2)
An effective stopping test must fulfill two conditions. First, it must have high stopping probability if the treatment harms any group of participants. Second, it must limit the probability of stopping if no group of participants is harmed (i.e., limit the "rate of unnecessary stopping"). Note that these conditions are equivalent to high statistical power and low Type error rate relative to (2). For the rest of the paper, we use these two metries- the stopping probability if any group is harmed and the rate of unnecessary stopping- to evaluate stopping tests with heterogencous populations.
Fig. demonstrates the limitations of applying stopping tests homogeneously to heterogeneous populations. We simulate an experiment that runs for 1,000 time steps and recruits 2,000 participants. Participants come from two groups, with G E {0, 1} indicating membership minority group. The treatment has no effect on the majority group but harms the minority with treatment effect 0, with NNO0 x D x G, o²) and o² = 1. The trial has one checkpoint the halfway stage and uses an OF-adjusted z-test to decide whether to stop for harm. We vary 0 from 0 to and (P = 1) from 0.1 0.5, and plot the stopping probability across 1,000 replications. We find that as the size of the harmed group decreases. stopping probability falls significantly. Continuing our warfarin example from Sec. if elderly patients comprise 10% of trial participants (lowest line in Fig. 1), then the stopping probability is less than 20%, even if the treatment has a large harmful effect (0 1).
3 Causal Latent Analysis for Stopping Heterogeneously (CLASH)
Thus far, we have introduced early stopping for harm and established that existing methods applied homogeneously may not be effective when applied to heterogeneous populations. In this section, we first develop useful intuition on heterogeneous early stopping. We then present CLASH, our new two-stage approach. Finally, we theoretically establish that CLASH improves stopping probability if the treatment harms any group of participants without inflating the rate of unnecessary stopping.
3.1 Motivation
The key problem with the homogeneous approach is that it averages over potential heterogeneous groups. One solution to this problem run stopping test separately on different subsets of the population, but a question remains: which subset guarantees a high stopping probability? Assume for a moment that we have prior knowledge of each participant's group Si and their CATE T(g:). Then, Prop. 3.1 (proved in App. I) establishes that for sufficiently large, the stopping test that only considers participants from harmed groups stops with probability approaching 1.
Proposition 3.1 (Group knowledge improves early stopping). Let (((((1))7(01) 0) denote the ATE for harmed groups and S (i < n T(9:) > 0} the participants from harmed groups. Let >; = /o denote the value of a test statistic computed only on S where EA; μ², - 4, (N 1). VSaion 1. o², and 18h diverges in probability to infinity. Fix any stopping threshold b(ai). Then, the stopping test that only considers S; has stopping probability converging to 1, i.e., P P(,; ; > b(a)) + 1.
The assumptions of Prop. 3.1 hold for many popular frequentist hypothesis tests, including an ad-ajjutted z-test, OF-adjusted t-test, and an OF-adjusted stopping test to detect differences in restricted mean survival time (RMST) for time-to-event data [28]. To give a rough intuition of
'W do not refer to these as statistical power and Type I error rate to prevent confusion between the hypothesis test of the overall experiment and that of the stopping test. However, note the equivalence.

---

the proof, the assumed conditions imply that the mean of x grows with [S;] while its variance remains bounded, and so X, eventually exceeds any fixed bound. Prop. 3.1 establishes that to stop the experiment with high probability, the stopping test should be run only on the set of harmed groups S We emphasize that the same stopping test applied to the whole population has no such guarantees. For example, if the overall ATE is O, then the stopping probability of the test applied homogeneously is < a. Thus, running the stopping test only on S, stops the trial with far higher probability.
One way to apply stopping test only to the harmed groups is to assign a weight to each observation " 1 Sü then reweight the test statistic by this indicator. For example, for the Od-adjusted z z-test in (1), using the following weighted test statistic is equivalent to running the test only on S
X; OF
VET V/2²2
yid. wid;
D (io((1-d1) Ei= (1-)),
(3)
Prop. 3.1 implies that using A, OF leads to high stopping probability. We thus call w the optimal weights, as using them reweight A, guarantees stopping probability close to for large enough
Of course, this insight is not of direct practical use, as it assumes knowledge of T(g)) and g. neither of which is observed. However, it motivates how to construct better stopping tests for heterogeneous populations. If we can infer from the data which groups are harmed, then we can use this information to accelerate early stopping. Specifically, if we can estimate the optimal weights wy,, then can these estimates to construct weighted tests that have high stopping probability. In contrast, stopping tests applied homogeneously will have no such guarantees. This is the insight that drives our method CLASH, which we present in the following section.
3.2 Method
At any interim checkpoint n. CLASH operates in two stages. In Stage 1, CLASH uses causal machine learning to estimate the probability that each participant is harmed by the treatment. Then in Stage 2, it uses these inferred probabilities to reweight the test statistic of any chosen stopping test.
Stage 1: Estimate w We first estimate the treatment effect (xx;) for each participant i S n using any method of heterogeneous treatment effect estimation. The specific choice of method is left to the investigator; potential options include linear models, causal forests [38]. and meta-learners [17].
To maintain conditional independence between the estimated CATE and the observed outcome for participant i, exclude (Ir; from the training set when estimating (a::). We thus use (x:)) to denote the estimated CATE for participant i, and (x;) for the estimated standard error of
(2,) (further discussed below). While the notation suggests leave-one cross validation, the estimation can instead use k-fold or progressive cross- vols-vvaiio [5]. We then set
4 = 1 - + ( (0_)._-((..) Gn. z(xi)
(4)
where & is the cumulative distribution function (CDF) of the standard normal and the hyperparameter
5 >> 0 (discussed below) is a small number. Intuitively, vi conveys the probability that i was harmed by the treatment; the choice of & in Sec. 3.3.
establish that it is good estimate of the optimal wi in See. 3.3. We also discuss
Stage 2: Weighted Early Stopping We now use wp to weight the test statistic of any existing stopping test. The choice of test is left investigators based on their experiment and domain. For example, CLASH with the OF-adjusted z-test would use the test statistic from (3), replacing w with vi. We provide examples of weighted test statistics for other stopping tests in App. C.
We make three important practical notes. First, cross-validation (CV) in Stage 1 is employed to prevent unnecessary stopping with small samples. CV ensures that if the treatment has no effect, then the predicted CATE for participant i will be independent of its observed outcome. Without this independence, outcomes that are large by random chance could be assigned high weights, especially in small samples. Second, most CATE estimation methods are designed to correct for confounding, which is not required in the experimental setting since the treatment is randomly administered. This simplifies the estimation problem (see App. D). Third, the CLASH weights (4) require not only point estimates of the CATE for each participant, but also standard errors. While bootstrapping can be used to obtain standard errors for any CATE estimation method, it is often computationally expensive. To save time and resources, investigators can use methods that compute standard errors during the process of fitting (c.g., causal forests or linear models).
5

---

Finally, we briefly discuss how to set the hyperparameter d. For our theoretical guarantees to hold (Sec. 3.3), 8 must be set to a number smaller than the smallest harmful treatment effect. This choice ensures that ri." converges to W for all values of Tzz)). In practice, we find that it is sufficient to set 5 to a number that reflects the experiment's minimum effect size of interest. is standard for clinical trials and A/B tests to specify this effect size, especially for pre-hoc power calculations [34]. Thus, ò can be set using the standard experimental procedures.
3.3 Properties
Recall that an effective method of heterogeneous early stopping must 1) have high stopping probability if any group of participants harmed and 2) limit the rate of unnecessary stopping. CLASH achieves both of these criteria by constructing weights uē that converge in probability to the optimal weights " A (Thm. 3.2). Thus, for n large enough, CLASH is comparable a stopping test run only on the harmed population S%, and hence has high stopping probability in the presence of harm (Prop. 3.1). Further, if no group is harmed, the CLASH weights converge quickly to zero. Hence, the test statistic used in Stage 2 converges to zero, as does the probability of unnecessary stopping (Thm. 3.3). Thus, CLASH has high statistical power and limits the Type error rate relative to the null hypothesis (2).
Our first theorem (proved in App. E.2) establishes the convergence of the CLASH weights to the optimal weights. It makes mild assumptions on the quality of CATE estimation in Stage 1. The choice of CATE estimator may demand further assumptions on the data; for example, using a linear regression provides suitable estimates if (x) is linear in z. Alternative causal machine learning methods minimize these assumptions: for example, causal forest provides sufficiently accurate estimates under weak regularity conditions (e.g., bounded [38, Thm. 4.1)).
Theorem 3.2 (CLASH weights converge optimal weights). CLASH weights (4) satisfy the error bound
|000 - w/] S exp {
(_()))-6)") ) 26 (z.)) S + [i (())))))( exp { (T(())-))* 26 ((a))
+ |=(z,)- ((x))))
(144)
(
Moreover, i inf (x)) and, given Ty, (x.) 4 (xx:) and (in)) B O, then oy is consistent estimator of the optimal weight: 1ff - wx 2, 0.
Thm. 3.2 provides important intuition on when CLASH will perform well. For observation i, the rate of weight convergence depends on two factors. First, it is proportional to (xx;). the size of the harmful effect. The more harmful the treatment, the better CLASH will perform. Second, it inversely proportional to ân, o(;),(x;), the uncertainty in the estimate of (2;)). The more certain the estimate, the better CLASH will perform. Many factors impact (z:). including n and the frequency of observing covariates like zi. We expect CLASH to perform better with (1) larger datasets, (2) fewer covariates, and (3) larger S,, We explore these relationships further in our experiments (Sec. 4-5).
We emphasize that Thm. 3.2 implics a fast rate of convergence, as the error bound decays exponentially in -/ For example, if (z;) = (1/ log n)), then iif - wi1 = op (1/m) (see Cor. F.1). This is extremely fast: weights and other nuisance parameters are often estimated at a rate of 1/ V or slower [6]. This speed justifies our use of the Gaussian CDF in (4). While other CDFs can provide Thm. 3.2's consistency result, they may not demonstrate such provably fast convergence. Note that the additional assumption on (x:) is mild,* as it just needs to decay faster than 1/ log For example, linear regression under standard assumptions (as in [36]) has (x;) = (1/v/nn).
Our next theorem (proved in App. E.3) establishes that CLASH limits the rate of unnecessary stopping for the OF-adjusted z-test. If no group of participants is harmed, CLASH's stopping probability converges to zero The rate of unnecessary stopping (i.e., Type I error rate relative to (2)) is thus guaranteed to be < for large enough. The theorem requires a mild condition on the decay of maxisn (z;) (the same discussed above). It also assumes bounded outcomes, a condition satisfied in most real experiments. Thm. F.2 provides similar result for SPRT stopping test.
Theorem 3.3 (CLASH limits unnecessary stopping). Consider a stopping lest with weighted z-statistic (3) and weights estimated using CLASH. If maxign 02 )2,-((a)) = op(1/ log(n)), (,))), and y are uniformly bounded, then the stopping probability of the test converges to zero if no participant group is harmed.
max;5n (Tx;)) - (2,) =
"See [6] for discussion of the rate of convergence of machine learning-base based CATE estimators.

---

We note that our presented theory is asymptotic, not exact. However, many stopping tests both frequentist and sequential-rehh on asymptotic approximations and so large enough sample sizes are often already required. For example, the mSPRT stopping test with binary outcomes in [14] requires a large-sample Gaussian approximation. In our simulation experiments (Sec. 4) and real-world application (Sec. 5), we show that sample sizes typical in many clinical trials and A/B tests are sufficient for CLASH to improve stopping probability while limiting unnecessary stopping.
3.4 The Decision to Stop
Before proceeding to our experimental evaluation, we emphasize that CLASH is not intended to automate human decisions on early stopping. When CLASH indicates that an experiment should be stopped, the investigator can make one of several decisions. If the nature of harm serious (e.g.. mortality in clinical trial), the investigator may decide to stop the experiment for all participants. For milder harms (e.g., high latency during application startup in an A/B test), they may decide stop the experiment only for the harmed group and continue it for all other participants. If the identified harm much less consequential than the potential benefit (e.g., harm of increased headaches vs. benefit of curing cancer), the investigator may decide to not stop the experiment at all. The specific decision must depend heavily on the ethical and financial aspects of the experiment, a thorough review of the interim data, guidance from the investigator's is institutional review board, and other relevant factors.
If the investigator chooses to stop the experiment only on the harmed group, they face two practical challenges: identifying the harmed group and ATE estimation. To identifyy the harmed group, investigators can either use existing subgroup identification methods (e.g., [1, 30, 44]) on the observed outcomes or analyze the CLASH weights to find covariate combinations with weights close to 1. To estimate the whole population ATE at the end of the experiment, investigators can use inverse propensity weights; this approach allows one to correct for the selection bias induced by stopping the experiment only in one group (see example in Tab. S6). Further details, including ensuring actionable group sizes, are discussed in App. G.
4 Simulation Experiments
We now assess the performance of CLASH in an extensive series of simulation experiments. 5 We focus on two important types of experimental outcomes; Gaussian and time-to-event (TTE). Gaussian outcomes are the most frequently considered (e.g., [14]), as they are common in both clinical trials and A/B tests. Many stopping tests assume either that the data is Gaussian or that the average outcome is usymptotically Gaussian. Meanwhile, TTE outcomes are very common in clinical trials. We establish that CLASH improves stopping times over relevant baselines in both these scenarios. Our real-world application (Sec. 5) then applies CLASH to outcomes that are approximately negative binomial, further demonstrating that CLASH can be impactful in experiments across several domains.
Setup: Gaussian Outcomes We consider randomized experiment that evaluates the effect of D on Y. Participants come from two groups, with group. We do not observe G, but observe set of binary covariates X [X1,.., Xa), where d varies between 3 and 10 and p(X, = 1) = 0.5 V j. X maps deterministically G. with = I Xy. We vary & between 1 and the expected size of the minority thus varies between 12.5% and 50% (of all participants). Y is normally distributed within each group, with Y|G 0 ~ ((, 1) and Y|G - (0, D. 1). The distribution of Y over the whole population is thus mixture of two Gaussians. We vary fu between O and 0011; the majority group is thus unaffected or benefited by the treatment. We vary 0, between 0 and 1: the minority group thus either unaffected or harmed.
{0, 13 indicating membership in a minority
The experiment runs for N = 4000 participants and T = 2000 time steps, recruiting one treated and one untreated participant each step." 7 The experiment has three checkpoints, with 1,000, 2,000, and 3,000 participants (corresponding to 25%, 50%, and 75% of the total time duration). At each
RRplicattion code for the simulation experiments can be found at https: / homh./////m/mmmmaaaammmm clash
'This is realistic setting (e.g., X is device type and new feature 1) increases application latency Y only on some devices). However, CLASH also performs well if the mapping from
is stochastic (see App. H.1.
"This assumption is required for some stopping tests (e.g., mSPRT, see [14]), not for CLASH. CLASH can be used with any experiment design (e.g. batch recruitment, imbalanced arms) if suitable stopping test is used.
7

---

O'Brien-Fleming
mSPRT
Mox5PRT
oracle that Bayesian Estimation
0.8
0.7
08
8
8 0.2
0.
0.
Lo
SHR ANI
8 0.8
0.3
0.4
0.2
0.
03
03
04
0.6
06
1.5
8.
02
0.4
06
08
1.0
no
05
04
06
088
1.0
0.0
0.2
0.4
0.6
03
1/
Harmful EreatrentEffector Minority Group
Oracie
CLASH
SUBTLE
Homogeneous
Figure 2: Performance of CLASH in simulation experiment with Gaussian outcomes. If the minority group is harmed, CLASH (red) significantly increases the stopping probability over the homogeneous baseline (blue) and SUBTLE (pink). For large effect sizes, CLASH is effective has prior knowledge of the harmed group (green). Crucially, if the treatment has no harmful effect (i.e., effect = 0), CLASH does not stop the trial more often than either baseline (i.e.. CLASH limits Type I error rate). These results hold for four commonly-use column). SUBTLE builds upon the mSPRT framework specifically, is only presented in the second column. These results hold whether the majority group's 's treatment is beneficial (top row) or has no effect (bottom row). We simulate 1,000 trials with 4,000 participants each and a 12.5% minority group: we plot the stopping probability (with 95% CIs) at any interim checkpoint.
stopping tests (corresponding to each
checkpoint, CLASH and compute its stopping probability across 1,000 replications. In Stage 1, CLASH uses a causal forest with 5-fold CV and 6 = 0.1. Standard errors were estimated by the causal forest itself (i.e., not via the bootstrap). In Stage 2, CLASH uses one of four commonly stopping tests: an -F-adjusted z-test, an mSPRT [14], a MaxSPRT [16], and Bayesian estimation-basse test.
We compare CLASH's stopping probability to three alternative approaches (two baselines and one oracle). The homogencous baseline applies the four aforementioned stopping tests to the collected data in aggregate. The SUBTLE baseline uses a SUBTLE the only existing approach that can handle unknown groups without strong parametric assumptions (c.g., linearity). SUBTLE builds the mSPRT framework and has the same decision rule; we thus compare its performance to CLASH using mSPRT in Stage 2. four aforementioned stopping tests only data from the harmed group. This approach reflects the optimal test discussed in Prop. 3.1, and represents an upper bound on performance.
heterogeneous stopping test [43].
The Oracle applics the
Setup: TTE Outcomes We consider a clinical trial that measures time to positive event (e.g.. remission) and defines treatment effects using the hazard ratio. We adapt the simulation setup from [20]; see App. 1.2.1 for full description. CLASH's Stage I uses a survival causal forest [8] and Stage uses an OF-adjusted Cox regression. Note that existing SPRTs cannot be applied to TTE outcomes, they cannot account for repeated observations for each participant. Crucially, SUBTLE cannot be used either; CLASH is thus the first heterogencous method applicable to this setting.
Results CLASH outperforms both the homogeneous baseline and SUBTLE in wide range of experiments. We first discuss performance in the Gaussian setting with five covariates and a minority group that represents 12.5% of participants. Fig. 2 plots the probability that the experiment is stopped at any of the three checkpoints. If the minority group harmed (x axis 0: > 0). CLASH stops the experiment significantly more often than the homogeneous and SUBTLE baselines. The magnitude of the improvement depends on the effect size: the larger the effect, the better CLASH performs. For large effects, CLASH even converges to the oracle. Crucially, CLASH does not increase the rate of unnecessary stopping: if the minority group unharmed (0, = 0), CLASH stops the experiment no more frequently than either bascline (i.c., CLASH controls the Type I error rate). These results
8

---

hold for all four stopping tests; CLASH thus leads to effective stopping no matter which test is used. Moreover, its performance gains are robust to the specific choice of the hyperparameter (Fig. S5).
We now compare CLASH to SUBTLE, the heterogeneous baseline presented in the second column of Fig. 2. CLASH improves stopping probability over SUBTLE whether the treatment benefits the majority group or has no effect. We focus on the former case and plot the increase in stopping probability of CLASH (using mSPRT) over SUBTLE in Fig. 3. CLASH improves stopping probability in large number of scenarios. We make three specific notes. First, when the minority group is larger, CLASH performs well for broader range of effect sizes. Notably, SUBTLE greatly underperforms CLASH for medium effects (though it is able to detect large effects as often as CLASH). Second, increasing the number of covariates only has a small impact on CLASH, which outperforms SUBTLE despite the higher dimensionality. Third, CLASH greatly increases the probability of stopping the trial the first and second checkpoints. CLASH thus not only stops trials more often, but also faster.
CLASH also demonstrates strong performance with TTE outcomes (App. H.2). CLASH improves the stopping probability of the OF-adjusted Cox regression by 20 percentage points if the treatment harms 25% of the population with hazard ratio of 0.7. No existing method for heterogeneous stopping (including SUBTLE) can handle TTE; this broad applicability one of CLASH's major advantages.
App. H presents detailed simulation results that further demonstrate CLASH's s efficacy. We note the discussed results robust across range of simulation settings, including effect size, sample size, covariate dimensionality, harmed group size, and outcome variance. Furthermore, CLASH is highly effective in situations with multiple harmed groups, noisy group membership, and unknown outcome variance. While CLASH broadly outperforms existing methods, we note two specific limitations. First, CLASH may struggle to detect harmful effects in very small samples (e.g, N=200, see Fig. S10). Note that this is a challenging setting for any method, since we only observe 10-20 data points from the harmed group any interim checkpoint. Second, CLASH may struggle with a very large number of covariates (e.g.. 500 covariates, see Fig. S11). In such settings, we recommend performing feature selection before running CLASH.
5 Real-world Application
In this section, we apply CLASH to real data from a digital experiment. We consider an A/B test run by a technology company evaluate the effect of software update on user experience. The dataset was collected in 2022 and consists of 500,000 participants, roughly half of whom received the update.
Checspein
Checkpoot
Checkpoat
Ehectporr
Eleckpoar
Checkpear
00
02
00
00
30
50 04 06 17
00 0.600000 60 08
64 10 Harmuu Treatment Effect on Minarity Group
900 04 0.6 10 00 1.
Harmful Treatment E fllct on Minarity Group
(a) Minority Group Size: 12.5%
(b) Minority Group Size: 25%
Figure 3: CLASH's performance improvement the SUBTLE baseline with Gaussian outcomes. CLASH significantly increases stopping probability over SUBTLE across a range of effect sizes (x- axis) and covariate dimensions (rows). CLASH not only stops stopping probability at earlier checkpoints (columns). CLASH does not increase stopping probability if the treatment not harmful (i.e., effect = 0). We plot the difference in stopping probability between CLASH and SUBTLE (with 95% Cls), where the minority group forms (a) 12.5% or (b) 25% of all participants. For the larger group size, CLASH's improvement most notable for medium effects, which SUBTLE struggles to detect. All performance increases are robust to an increase in the number of covariates. See App. H.l for a similar comparison of CLASH the homogeneous baseline.
often but also faster, with higher

---

We evaluate this treatment's 's effect on a proprietary metric that measures negative user experience. An increase in the outcome-whiee whhch is discrete and hight-skewed-indieee
worse user experience. The experiment also recorded which one of eight geographic regions the user's device was located in. We first use the full dataset to assess whether the treatment had a heterogeneous impact by region. Separate negative binomial (NB) regressions indicate that the treatment led to large statistically significant increase in the metric in Region I, a small significant increase in Regions 2, 3, and 4, no significant effect in Regions 5 and 6, and significant decrease in Regions 7 and (Tab. S3).
We evaluate stopping tests on this dataset, conducting checkpoints every 20,000 participants. We run CLASH with a causal forest and 8 = 1 in Stage I and an OF-adjusted NB regression in Stage 2. We find that CLASH decreases stopping time over the homogeneous baseline, stopping the experiment after 40,000 as opposed to 60,000 participants. SUBTLE stops after 80,000 participants; this under-performance may result from the data's strong non-normality. An oracle that only considers Region 1 also stops after 40,000 observations (i.e., same as CLASH). We note that CLASH's optimal performance is not fluke: across 1, 1,000 random shuffles of the dataset, CLASH stops the experiment at the same interim checkpoint as the Oracle in 62.6% of shuffles (details in Tab. S5).
We emphasize that by stopping the experiment early, the company can minimize customer frustration, preventing churn and financial impact. Domain expertise can guide whether to stop the experiment altogether, or just in the harmed regions. The harmed regions can be identified at stopping time using separate regressions (Tab. S4), CLASH weights (Fig. S19), or policy learning [15, 1]. Note that stopping the experiment just in one region would affect statistical inference at the end of the experiment, as the treatment would no longer be randomly assigned across regions. As described in Sec. 3.4, investigators can use inverse probability weighting to adjust for this selection (Tab. S6).
Finally, we study the impact of sample size on CLASH's performance. We vary sample size N, between 10,000 and 100,000, and gener- ate 1,000 experiments for each N, by sampling from our dataset. We only sample from Region 1 and Regions 5-8: this gives us one harmed group (Region 1) that comprises 29% of the total population (see App. for results with Regions 2-4). We conduct one checkpoint at the halfway stage of each simulated experiment. We find that CLASH significantly improves stopping prob- ability over the homogeneous baseline for all
1.0
Stopping Probabilty
0.9
0.8
0.7
0.6
0.5
0.4
0.3
10x
20k
30K 4bx 50k
60K
76x sbk 90K 100K
Size of Epperiment (Ng)
Oracle
CLASH
Homogeneous
N. 2 40,000, and converges to near-oracle per- formance around N = 90,000 (Fig. 4). These sizes represent 8% and 18% of the total sample of the A/B test. This result is encouraging, as indicates that datasets that are fraction of the size of typical online A/B tests are large enough for our asymptotic theory (Sec. 3.3) to hold.
Figure 4: CLASH performance by sample size on real data. We plot stopping probability across 1,000 experiments with 95% CIs.
6 Discussion and Limitations
We propose new method CLASH for the early stopping of randomized experiments on heterogeneous populations. CLASH stops experiments more often than existing approaches if the treatment harms only a subset of participants, and does not stop experiments if the treatment is not harmful. Prior work is either limited by restrictive assumptions (e.g., linearity) or incompatible with common data distributions (c.g., time-to-event). In contrast, CLASH easy to use, adapts any existing stopping test, and is broadly applicable to clinical trials and A/B tests. Our work has few important limitations. First, CLASH works better with a relatively small number of covariates. While this is not a problem for most experiments clinnca trials and A/B tests typically collect only a few covariates a different pre- pre-processing approach may be required in nigh-iimeniional settings (e.g., genetic data with over 500 covariates). Second, CLASH may not detect harm in experiments with a very small number of participants. For example, sample sizes typical in Phase 1 clinical trials (i.e., less than 100 participants) may be too small for CLASH to be more effective than existing approaches. Finally, this paper only considers stopping for harm. In practice, experiments are often stopped for early signs of benefit and futility. Expanding CLASH to these decisions is an important direction for future work.
10

---

Acknowledgments and Disclosure of Funding
We thank Steven Goodman, Sanjay Basu, Dean Eckles, Nikos Trichakis, and Sonia Jaffe for their helpful comments. This work was graciously supported through the Bowers CIS Strategic Partnership Program with LinkedIn.
References
[1] Susan Athey and Stefan Wager. Policy learning with observational data. Econometrica, 89(1): 133-161, 2021.
[2] Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests. The Annals of Statistics, 148-1114, 2019.
[3] Akshay Balsubramani. Sharp finite-time iterated-logarithm martingale concentration. arXiv preprint arXiv:1405.26339 2014.
[4] Donald A Berry. Bayesian approaches for comparative effectiveness research. Clinical Trials, (0:37-47,,001.
[5] Avrim Blum, Adam Kalai, and John Langford. Beating the hold-out: Bounds for k-fold and progressive cross-validation In Proceedings of the twelfth annual conference on Computational learning theory, pages 203-208, 1999.
[6] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, and Whitney Newey. Double/debiaseddnemma machine learning of treatment effects. American Economic Review, 107(5))261-65, 2017.
[7] Thomas Cook and Olive D Buhule. Stopping trials early due to harm. NEJM Evidence, 1(5): EVIDetw2100026, 2022.
[8] Yifan Cui, Michael R Kosorok, Erik Sverdrup, Stefan Wager, and Ruoqing Zhu. Estimating heterogeneous treatment effects with right-censored data via causal survival forests. arXiv preprint arXiv:2001.09888, 2020.
[9] Richard E Deichmann, Marie Krousel-Wood and Joseph Breault. Bioethies in practice: consid- erations for stopping clinical trial early. Ochsner Journal, 16(3):197-198, 2016.
[10] David L Demets and KK Gordon Lan. Interim analysis: the alpha spending function approach. Statistics in Medicine,
1994.
[11] Alex Deng, Jiannan Lu, and Shouyuan Chen. Continuous monitoring of A/B tests without pain: Optional stopping bayesian testing. In 2016 IEEE international conference on data science and advanced analytics (DSAA), pages 243-252. IEEE, 2016.
[12] Christiana Drake. Effects of misspecification of the propensity score on estimators of treatment effect. Biometrics, pages 1231-1236, 1993.
[13] Steven Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055-1080 2021.
[14] Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Peeking at A/B tests: Why it matters, and what to do about it. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1517-1525, 2017.
[15] Toru Kitagawa and Aleksey Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica, 86(2):591-616, 2018.
[16] Martin Kulldorff, Robert L. Davis, Margarette Kolezak, Edwin Lewis, Tracy Lieu, and Richard Platt. maximized sequential probability ratio test for drug and vaccine safety surveillance. Sequential analysis, 30(1):58-78, 2011.

---

[17] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 166(10)4415-44165 2019.
[18] John M Lachin. A review of methods for futility stopping based on conditional power. Statistics in Medicine, 24(18):7747-2764 2005.
[19] Gary Lorden. 2-sprt's and the modified kiefer- weiss- problem of minimizing an expected sample size. The Annals of Statistics, pages 281-291, 1976.
weiss
[20] Ying Lu and Lu Tian. Statistical considerations for sequential analysis of the restricted mean survival time for randomized clinical trials. Statistics in biopharnaceutical research, 13(2): 210-218, 2021.
[21] Xinkum Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-31, 2021.
[22] Peter C O'Brien and Thomas Fleming. A multiple testing procedure for clinical trials. Biometrics, pages 549-556, 1979.
[23] Stuart J Pocock. Group sequential methods in the design and analysis of clinical trials. Biometrika, 64(2):191-199 1977.
[24] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2022. URL https: //t/://wwwww R- project. org/.
[25] Aaditya Ramdas, Johannes Ruf, Martin Larsson, and Wouter Koolen. Admissible anytime-valle sequential inference must rely on nonnegative martingales. arXiv preprint arXiv:2009.03166, 2020.
[26] Herbert Robbins. Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics, 41(5):137711409 1970.
[27] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):8468666 1994.
[28] Patrick Royston and Mahesh KB Parmar. Restricted mean survival time: alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome. BMC medical research methodology. 13(1):1-15, 2013.
[29] Benjamin R Saville, Jason T Connor, Gregory D Ayers, and JoAnn Alvarez. The utility of bayesian predictive probabilities for interim monitoring of clinical trials. Clinical Trials, 11(4): 485 493, 2014.
[30] Patrick M Schnell, Qi Tang, Walter W Offen, and Bradley Carlin. A bayesian credible subgroups approach to identifying patient subgroups with positive treatment effects. Biometrics, 1226 2016.
[31] Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
[32] Aditi Shendre, Gaurav M Parmar, Chrisly Dillon, Timothy Mark Beasley, and Nita A Limdi. Influence of age on warfarin dose, anticoagulation control, and risk of hemorrhage. Pharma- cotherapy: The Journal of Human Pharmacology and Drug Therapy, 38(6):588-55) 2018.
[33] Chris Stucchio. Bayesian A/B testing at VWO. Whitepaper. Visual Website Optimizer, 2015.
[34] Gail M Sullivan and Richard Feinn. Using effect size or why the p value is not enough. Journal of graduate medical education, 4(3):299-82,, 2012.
[35] Peter Thall and Kyle Wathen. Bayesian designs to account for patient heterogeneity in phase ii clinical trials. Current opinion in oncology. 20(4):407, 2008.
[36] Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge University Press, 2000.
12

---

[37] Basil Varkey. Principles of clinical ethics and their application to practice. Medical Principles and Practice, 30(1):17-28, 2021.
[38] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113652))12288-1222 2018.
[39] Wald. Sequential tests statistical hypotheses. Annals of Mathematical Statistics, 1945.
[40] Ian Waudhy-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. arXiv preprint (2Xi::2010009686 2020.
[41] Liuyi Yao, Zhixuan Chu, Sheng Li, Yaliang Li, Jing Gao, and Aidong Zhang. survey on causal inference. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(5):1-46, 2021.
[42] Miao Yu, Wenbin Lu, and Rui Song. new framework for online testing of heterogeneous treatment effect. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 00100-0317, 2020.
[43] Miao Yu, Wenbin Lu, and Rui Song. Online testing of subgroup treatment effects based on value difference. In 2021 IEEE International Conference Data Mining (ICDM), pages 1463-1468. IEEE, 2021.
[44] Zhongheng Zhang, Heidi Seibold, Mario V Vettore, Woo-Jung Song, and Vieille François. Subgroup identificatinn in clinical trials: an overview available methods and their implemen- tations with r. Annals of translational medicine, 6(7), 2018.
13

---

A Early Stopping Methods
Table S1: Comparison of stopping tests. We categorize commonly stopping tests into three categories Eneegorees-frqqeeee frequentist, sequential, and Bayesian- and describe their key features. Stopping type describes whether the stopping test is used to stop for benefit (B), harm (H), futility (F). Stopping criteria refers to the statistic that is used to make the stopping decision. Monitoring frequency describes how often the method allows peeking the data; for example, frequentist methods only allow for a small number of interim checkpoints, while sequential methods allow for continuously monitoring outcomes as they are observed. Finally, the test type describes what kinds of alternate hypotheses the method allows for testing; the SPRT, for example, only allows the investigator to test simple vs. simple hypotheses.
Stopping Test
Stopping Type
Stopping Criteria
Monitoring Frequency
Test Types
Frequentist / group sequential
Pocuck [23]
v-vauue
Limited checks, pre planned
[22]
B/H B/H
p-value
Limited checks, pre pre-planned
All
Alpha-spending [10]
p-value
Limited checks, fletible
Conditional power [18]
F
Conditional power
Limited checks. flexible
All
Sequentall quasá Bayesian
SPRT [39]
B/H/F
Likelibood ratin
Continnous
Simple
2-SPRT [19]
B/H/F
Likelihood ratio
Continuous
sidel
Max-SPRT [16]
B/H
Likelihood ratin
Continuous
Two-sided
Mixture-SPRT [26, 14]
B/H
Likelihood ratio always-valid p-value
Continuous
Two--sded
Martingale-bused [3, 13]
Always-valid p-value
Continuous
Bayesian
Purameter estimation [33]
Posteriar expected
Continuous
Hypothesis testing [I]]
B/H
Posterior odds
Continuous
All
Prodictive probability [29]
F
Posterior predictive probability
Continuous
B Adapting the SPRT for Early Stopping
Here, we briefly describe how to adapt SPRT-based tests for early stopping. This largely follows from Section 4.3 in [14]. Recall that for every participant we observe their treatment assignment dj, outcome yi, and covariates We now slightly update this notation. Assume that at time I, we observe two participants, one treated, one untreated. Let yi
(1) and yt (0) denote the outcomes for the treated and untreated participants respectively, and denote their joint covariates. We define the (1)
difference in outcomes observed at time 2t Yt (
We can now use Wald's SPRT [39], mSPRT [14], and other sequential tests in this setup. For example, assume that follows a known distribution with probability density function p and mean parameter μ.. Then, to test between the null hypothesis of no effect Hy μ = 0 against an alternate hypothesis of specific harmful effect H1 / 8, Wald's SPRT would use
p(zelu = B)
N
SPRT = Σ log
p(a. = 0)
t=1
SPPR (a: = log a
Note that we denote the statistic A untreated). For the special case where p is the Gaussian probability density function, the test statistic reduces
SPRT not ASPRT it uses 2n observations (n treated, n
SPRT A2n
g2
21
t=1
14

---

C Weighted Test Statisties
We present examples of weighted test statistics for some commonly-used stopping tests in Tab. S2. Note that weighted inference is already implemented in standard data analysis packages for wide range of estimators. For example, for a group- -euueettitt test (e.g., O'Brien-Flee Frrien-Fleming) that considers p-values from generalized linear model, standard packages in R [24] and Python [31] allow for user-defined regression weights.
Table S2: Weighted test statistics for commonly used stopping tests,
Stopping Test
Test Statistic
Weighted Test Statistic
Group-sequential z-test
22
med Ei=: d
I (1-4))
(1-((1-)))
V25
Miseed mids
En mill (1)
win (1-(1
Generic SPRT
Ei-1 log p(zclu=3)
pla.la-0) Ei= We log
p(oela=d)
p(silMED)
Gaussian SPRT
Ei= 21
Σi=l 0/22
Eig Wh 182
Gaussian mSPRT
2nd 2x + tn exp
+ *II Bon) An (
2x 2n
exp
P Tut 2ar + 2irmm mi)
mast do Dim wz)
[14]
Gaussian maxSPRT
E 21m1
E
E
Eng my
Em Eo Meet
Le
(based on [16))
D CATE Estimation with No Confounding
Most CATE estimation methods are designed to control for confounders, that is, variables that affect both the treatment status D and observed outcome Y. Our work, however, considers experiments in which D is randomly administered; there is no confounding, as D is independent of all variables except Y. The CATE estimation method thus only needs infer how the treatment effect varies with the covariates X and does not need to correct for the relationship between X and D. This simplifies the application of several CATE estimation methods; we discuss few specific examples below. In short, many methods use estimates of the treatment propensity ()x)) = PPD 1|X = x) to correct for confounding. In our setting, we can simply set (7(2) (=)= 0.5 V
Causal forests While the original causal forest algorithm [38] did not estimate propen- sity score, later improvements [2, 21] use estimates of ()x) to "orthhoonalize"" the estimator. In practice, this orthogonalization is essential in obtaining accurate CATE estimates from ob- servational data [2]. However, it is not required in our experimental setting; these improved algorithms [2, 21] can be used by setting 7(x)) its true value (i.e., 0.5).
Meta-learners T-learners and S-learners (as described in [17]) do not use estimates of the propensity score and thus do not require any modifications. The X-learner [17] uses an estimate of 17(x)) to weight the predictions of models trained separately on the treatment and control groups [see 17, equation 9]. These weights can be replaced by 0.5 in our setting.
Linear Models Linear CATE models often use estimates of 1/n7æz) as regression weights. This approach ensures that the estimation "doubly robust": as long as either the model of treatment assignment or outcome heterogeneity is correctly specified, the resulting regression coefficients unbiased [27]. In our setting, such reweighting is unnecessary, as 1/n(æ) is known and the same for all observations.
Not needing to estimate the propensity score has two key benefits. First, it reduces the computational complexity of the problem. Repeated fitting techniques like cross-fitting [6] are often used to reduce the bias of simultaneous propensity score and treatment effect estimation, but these procedures increase the computational cost CATE estimation, especially for large datasets. Second, protects against inaccurate estimates of the treatment propensity. Poor estimates of 7π(z)) can lead to highly biased treatment effect estimates [12]. By explicitly specifying 17(x) = 0.5, we avoid spuriously inferring any relationships between X and D that would create such bias.
15

---

E
Proofs
We now present proofs for all theoretical results described in the main text. Note that we repeatedly the following property, For any events A and B,
P(A.B) & min (PAA),
(5)
E.1 Proof of Prop. 3.1: Group knowledge improves early stopping
We argue that the stopping probability of the test on S converges to 1. Recall that
By assumption, X /0 N(0, and 0 2 Define P. /3/0" Then, by Slutsky's theorem,
0 - V 4 (0 1).
Now, consider the probability that the test does not stop. For notational convenience, fix some and denote b(a) simply as b. Then,
P( b) P I bpp
P( S /p Nr PT/X; 2 PP 6pt P.IX, 0.5)
< Pp, I 6/2) + P(p < 0.5)
(6)
Now, we know that,
Ph
S /
V S o
2
Hence, (p75/A < 0.5) + 0. This bounds the second term in (6). We now focus on the first term
Fix any > 0. Since |S - co, there must exist some n, such that b/2 n1 VS < (c) for all n > ng. Assume n ne. Then,
P(hh < 6/2) Pp - μ² / b/
< Pp' n² VSV (e)
(c)) =c
since Ph, - 1¹¹ / 4 ((O 1). Since this holds for any value of we have that ((, S
6/2) + Hence, from (6), we have that IP (A; 6) and thus
P( 6) + 1
Hence, the stopping probability of the test on the harmed population converges to 1.
F.2 Proof of Thm. 3.2: CLASH weights converge to optimal weights
Our entire proof will be carried out conditional on the value 2;. Recall that define our weights as
+ Gn (I;)
Further recall that the functions 6, and on are, by construction, independent of 2;.
16

---

Error Bound We first establish a bound on the difference between our estimated weights v² and the optimal weights w. Consider two cases.
Case 1: =(x;) I O. In this case, W = O by definition, and so we just need to prove a bound on the magnitude of u... Using Taylor's theorem with Lagrange remainder, we know that Rh, [0, such that
0 - Fn, (axa)
v = =1- +
Ön,-i (2x)
8 T (3z))
r(x;)) (1a))
8 - (xx))
T(oi) - - (n,-)((a))
(z,) ②
+ hm
(2+)
(x;)
8 T (2is)
(x()) Fr, (xحح)
8 8- T(Ti)
T(u;) - fn, ((x1)) 8n, (111)
2 +
+ 1nn
(ni)
(x:)
where @ is the standard Gaussian probability density function. Since 8 - (xx) > 0. we can use the Chernoff inequality to bound the first term,
1 Φ 8 (241)
(- (6 ((=)
S exp
0 (2:)
26 20
We now focus on the second term, which contains
5 (1;)
T(x;) (xx))
hn
5 n. (I:)
18 - ((xi)
T(2r)) - pn, ((1)) 2
cxp
+ hn
v 2x
2 inz-
)n,)((I;)
0 - (1xs)
2
(133) t (15a) 2
V2T
exp
2
8 (x,)
+ +1
(8 T (Tx;)) (+n. (1x;) - T(zs))
2hn
- (1:)
T (1x))
2
(8 T(xx)) - (1;) - T(x:)1
S exp
2hh
2
i (x:;)
02 n (x;)
2 (6 ((x:)) n, - (3,) -(62))] }
1
8 T (2;
S exp
2
(x,))(x))
0² (zi)
since hn E [0, 1). Thus, we have that
oF S exp { (f-r(x;))² 24 (..)
+ (r(2))
(2=)( + (5 (5-r(x,)) (x=)) (21) (1,)
exp
(2,)
Case 2: (xx)) > 0. In this case, w²- 1. Thus,
|10f w.l
1 - Wt
Φ 5- An, (1(x))
o, (xx)
8 (2x))
(T(x)) - Fn, ((xi)
5 -
T(I;) 8m, -x,i)((i)
+
+
+ hnn
i (mts)
(xx)
~n,-((1))
an,-i(x;))
T (2x) - 8)
(T(2g) - Fn, (2x5)
- T(x:)
+ -n T(xs) - fn, (x22)
1 o
+
Gn (z;)
8n )n;)
0 (I;))
Ōn, i (2x)
(8- (2x;))2
< exp
202 n- 202_-(10) (2i)
(T(::) - Fxn-)(2;)
(6 - T(::))
(T(x.) - (1;:) - T (x:)
(1;))(1;))
exp
20 n, (1:) (1;)
+
02 - 1 (x:)
17

---

using the same Taylor expansion and Chernoff bound from above. In summary, we have established that
b:n - w S exp {
(8-7(x,))) 28 n, (a.)
[m:]] [F((,)) [m:]] exp
(8-
((an))6lltt... (at
28 2, (ma)
d (x,)
Consistency of w We now establish the consistency of uu using the derived bound. We assume that 8 < (s)s0 (()) and, given Ti. in, ix;- & (x;) and iz;)) 2 0.
Define a; = (xx)) - 81/V2 and Zi, z(z,) ((n)) Note that a, is strictly positive and constant (given 24). From the eπror bound, we have that
an. (()))
|00 - u S exp(- /=n_ + Iz axpp-a//ón, + /2u; 2iolon
Now, we fix any E > 0. Applying the law of total probability and equation (5), we find that
P( wil c)
SP(exp(- /
/ i (x:;) + (oppeepppp_/nn- /n >
=P(exp(-oo -
+ Zi.al - 2- 220; Zi. / [(x:) > n > 0:/ (2)2))
((exp(-a//0n- + [i,nl lann/eppla//// 14 >E,
(z;) a: (2v2))
< O > a (2v2))+
P (lexp(-a//on,--l/
+ ZZ, expl (m, (aao-/laa)(2222/./////ooo--- (200
(Zinonn, S a. / (2V2)
<P <P(iz,nnnn,i (x;)) > /: (2v2))+
PP
(1+
2 (0 (1:) ) exp
0i
at
>c, 1Z,n S a: (2V2)
28 -
(((Zz,nn.,((zz)) > (2/(222) + P ((+ 2v2ōn,-((xa)
a,
exp
20 - 20n,-i(2:)) )2
>
where the second-to- inequality follows by substituting -i S a./ (2v2) into the bound and algebraically simplifying, and the last inequality follows from (5). Now, the first term on the right converges to 0 since Zi (xx) (xx) - fn, (x;;) 2, O. We examine the second term. Define εr, (V20,, n, (xx,)). Since convex function is smaller than its tangent line, we have (1 + E2,n22 I exp(Ei.n/ 22). Then,
P (1+
2 (0
a;
exp
20 20 (x,)²
>
P(1 + Ex,n /s.l/))xp((--- 52,)) > E)
< P((epp(-Ee 2 + Ei,/2)) /2) > 6)
which converges to 0 as (x)))(x;) B (and thus Ei, diverges in probability to oo). Thus, we have shown that
P( w/ > c) + +0
which establishes the desired consistency.
E.3 Proof of Thm. 3.3: CLASH limits unnecessary stopping
Suppose that no participant group is harmed so that w 0 for all i. Leta = 8/ and a, = -[((;;)~ 61/ /2, and note that S a, for each as each T (2+) < 0. Thus, if (z, a2(2))), then
18

---

Zinn I (2/(2v2) and hence, as in the proof of Thm. 3.2,
xF - (1 (1+ 2v 2v20 n (z;)
a
exp
25 n, (24) (2;)3
Fix any let Ei.. -s/ - (2:)), and define f(x) (1 x/2) exp(-x²). As did for equation (7) in the proof of Thm. 3.2, we apply the law of total probability and equation (5) to write,
P P 5h > c) IP (max, sn Zi mi. > (2v2))
P(E, of > , maxisn n, S (2v2))
P (maxign
> (2v2)))
P (1 + a: (2 26 (z:)) exp( at (2 > c) / (2V2)) - P(E, ((i,n)) c).
=P (maxign Zi n, (m:) -
Since maxign n = (1) by assumption, we have P P(max;gn -n > a/(2/2) + 0. Meanwhile, since a convex function is no smaller than its tangent line, we have 2/+ < cxp(2/2) and hence
(x) exp(-=²++ x/2) < exp(-x> 22 exp(-
where we used the arithnetic-geometric mean inequality in the final inequality. Therefore, since a S min; ai
P.E, 2)) P(E; exp( (22
S exnexp( (mmmpppmminnss-.. 2/,)/2 1/8) > c)
I P(rcxp(-(minng. (4)2), 1/8) > c)
- P((epp(aa//( 1 + 1/8 + logn) > >e)
Since maxi5n o2 (x,) [[]]lo)(]))))))) by assumption, further have POE; f(Eim)) >)) +
Since 0 was arbitrary, have shown that Ei= uif 2, O.
Now, recall the form of the CLASH weighted z-statistic
Ei=1 up
Di=1 if yid, EL
Di- - d:)
A
22
o.d
Σ 60 d,)
Define such that 1/s S We know must exist, since the outcomes are bounded by assumption. Then, we have that
W yid
E 1 d4) m
200
21 1=1 w/ di
1=1 26 (1 - di)
1-1 i
uis edi
>m d (-c) di)
/202
1=1 w
Σ 1=1 w di)
Σ 1=
2c
22022
2
Thus, we see that the weighted test statistic A converges in probability 0. Now, the test can only reject if X exceeds a fixed and positive bound by. By the definition of convergence in probability. this probability must shrink to zero.
19

---

F Additional Theoretical Results
Corollary F.1 (CLASH weights converge quickly optimal weights), Assume that o² (x;) [o)]). Then, Thm. 3.2 implies that b w op(1/n).
Proof. Define o; = 1x(=;) 81/v2 and Zi, I(a;) ((aa))-nn--l Fr. -(a)) Fix any > 0, let Sin
(2,)
Then, in equation (7) in the proof of Thm. 3.2, we apply the law of total probability and equation (5) to yield
a:/(Von,.. (m:)), and define ((x) (1 + x/2) (=xp
P(n|0, =wl > e) S P( on -(x)) > a;((2v2))
+
a2
a
P n
+
2 20 n (z:g
exp
20 ( (1;)2
>6
ai (2 (2)2)) P
P(nf(E > >c)-
The first term the right converges to 0 since ni assumptions of Thm. 3.2). We focus on the second term. Since a convex function is no smaller than its tangent line, we have (1 + /2, exp(Ei,n/2) and hence
T(x,) fn, (x,) 2 0 (by the
(Ei.)) S exp(- + 22) S exp(- C +
exp(- 22 1/8)
where we used the arithmetic-geometrii mean inequality in the final inequality. Thus,
P(nf(EEnn) > e) < P(n (fnneex(((,,,,
e)
P(exp(-o0/400_- 1/8 log n) > E)
+ 0,
since -i = (p(1/og((n) Thus, (n] - c) + and so by definition, | w// (1/nn)
Theorem F2 (CLASH limits unnecessary stopping for the Gaussian SPRT). Consider a stop ping test with the Gaussian SPRT and weights estimated using CLASH. If max 6n.-1(23) op(1/ log(n)). maxtsn probability of the test converges zero if no participant group is harmed.
= o,(1), and z are uniformly bounded, then the stopping
Proof. We adopt the setup for the SPRT described in App. B. Recall that the Gaussian SPRT test statistic is given by
A2n
3 21
72
82
2
t=1
and the CLASH- weighted version of this statistic given by
Di=1 W B2
Am
3
2
2
Now, by assumption, there exists some e that bounds | for all 1. Since no participant is harmed,
w = 0 for all i. Similar to the proof in App. E.3, if max,5n (n,=1(x;)2 = (p(1/ ([//oo/)))) and max,5 (r(x,) An, (x,)) = (1). then " ui + O. Thus, since 2; are uniformly bounded by some c,
A
< 3
Σ
1=1
P 0
This shows that Awr &. 0 as + co. Thus, P > log + 0 for any fixed a > 0, proving the claim.
20

---

G Stopping Only on the Harmed Group
If the investigators choose to stop the experiment only on the harmed group, they face two practical challenges: identifying the harmed group and ATE estimation. We discuss each of these below.
Identifying the harmed group. To stop the experiment only for harmed participants, investigators must first identify the harmed group. This is non-trivial, since group membership is unobserved. The distribution of CLASH weights stopping time can help in this task: groups with estimated weights close to are likely to be harmed. Fig. S19 illustrates this in our empirical application: the estimated weights in Regions and are both close to I, indicating that these are the groups which to stop. With few covariates, investigators can manually inspect the weight distribution for each covariate combination to identify the harmed group. With many covariates. investigators can use a simple heuristic; a regression decision tree on the estimated CLASH weights can find the covariate values for which the weights are the largest. Limiting the depth of this tree can ensure that the identified group is actionable (i.e., it is possible to stop the experiment on it) and of non-trivial size.
There alternative approaches to this harmed group identification task; for example, investigators can use subgroup identification methods on the raw outcomes (e.g. [1, 30, 44]). We are agnostic to the choice of method: investigators can pick the approach most appropriate for their domain.
ATE estimation. Stopping the experiment on only one group can affect inference at the end of the experiment, the treatment is longer randomly assigned across covariates. To estimate the whole population ATE, we recommend using inverse propensity weights (IPW) to correct for the induced selection bias. Let ğ be the group that the experiment is stopped on and p(ğ) denote the proportion of the total population that comes from g. p(ğ) may either be known (e.g.. from domain knowledge prior experiments) estimated from the interim data when the experiment is stopped on ğ. Let ( denote the observed proportion of 9 in the data collected over the entire experiment (i.e., pre- and post-stopping I should be assigned a weight of p(g) Peet/tetted (g) and all other observations should be assigned a weight of 1. This IPW approach will lead to unbiased ATE inference (see the example in Tab. S6).
g). Then, to estimate the whole-population ATE, observations from
21

---

H Additional Simulation Experiments
In this section, we present detailed results from our simulation experiments. App. H.I presents results with Gaussian outcomes and App. H.2 presents results with time-to-event outcomes. In almost all settings, CLASH outperforms both the homogeneous and SUBTLE baselines. We note that CLASH's performance gains are relatively limited in three situations: (1) when the harmed group forms 50% of the population (i.e., no longer a true "minority") and the treatment has no effect on the remaining population, (2) when the experiment collects very large number of covariates (d = 500), and (3) when the experiment recruits very small number of participants (N 200). However, even in these settings, CLASH's stopping probability is equivalent the baselines.
H.1
Gaussian Outcomes
H.1.1
Setup
The simulation setup the same as described Sec. 4; we include it here again for easy reference. We consider a randomized experiment that evaluates the effect of D on Y. Participants come from two groups, with G E {0, 1} indicating membership
minority group. We do not observe G, but observe set of binary covariates X = [X1,.., Xa]. where d varies between 3 and 10 and p(X, = 1) 0.5 j. X maps deterministically G, with G I[; X,. We vary between 1 and 3: the expected size of the minority thus varies between 12.5% and 50% (of all participants). Y is normally distributed, with Y|G = ~ (,, 1) and Y ~ D, We vary Og between 0 and -0.1; the majority group is thus unaffected benefited by the treatment. We vary 0 between O and I: the minority group is thus either unaffected or harmed.
The experiment runs for N 4000 participants and T 2000 time steps, recruiting treated and one untreated participant at each step. The experiment has three checkpoints, with 1,000, 2,000, and 3,000 participants (corresponding 25%, 50%, and 75% of the total time duration). At each checkpoint, we run CLASH and compute its stopping probability across 1 1,000 replications. In Stage 1, CLASH uses causal forest with 5-fold CV and 5 0.1. Standard errors were estimated by the causal forest itself (i.e., not via the bootstrap). In Stage 2, CLASH uses one four commonly
stopping tests: OF-adjusted z-test, mSPRT [14], MaxSPRT [16], and Bayesian estimation- based test.
Each replication uses its own random seed. One replication-includiin all combinations of effect size, minority group size, number of covariates and stopping tests-takes between 58 and 82 minutes to run on a single CPU (depending on the random seed). This yields total compute time of approximately 1,200 hours for the Gaussian experiments. The simulations were run in parallel on academic computing cluster with over 200 CPUs. Note that each replication used one CPU with 4GB of RAM.
22

---

H.1.2 Impact of Hyperparameter 8
We first discuss the choice of hyperparameter 8. We recommend that investigators set this value to the minimum effect size interest (MESI). Here, we evaluate three different values of the MESI: 0.05, 0.1, and 0.2. Note that effect size of 0.05 is detectable in this experiment (with N = 4, 000) with 70% power; we thus not consider MESI 0.05, as such small effects would not be reliably detectable.
We present our results for CLASH with an OF-adjusted z-test and an mSPRT in Fig. S5. In most scenarios, all three settings of 6 perform similarly. For the mSPRT with a 12.5% minority group, we note that larger value ó corresponds with higher stopping probability for larger effects. This intuitive, as the higher 8 is, the more weight CLASH will assign to parts of the covariate space that show clear signs of harm. However, CLASH performs better than the homogeneous and SUBTLE baselines for all three values of 5. Thus, the results presented in the main text are robust to an increase or decrease in this hyperparameter.
Vi Firrrty Group Sow 1135
Miverly Group See 20%
L.
ns
05-
us
03
Enz
01-
Lo
Pos
05
0.4
0.34
02
0.1
Harmful Treatment E fiect Minorry Group
Oracle
CLASH (6 =0.2)
CLASH (8 (6:0.1)
CLASH 16 =0.05)
Homogeneous
(a) O' Brien Fleming
ranty Group Sie 1255
Minrts Group Sier
1.0
0.5
0.6
u5-
0.c
033
3 so0
P
0s
Suc
033
0.
04
00
02 Minoriy Group
0.6
Harmful
CLASH 16-011
SUBTLE Homogeneous
CLASH -0.051
SSPT
Figure $5: Rffect of CLASH's stopping probability. We consider the stopping probability of an (a) aF-adjusted z-test and (b) mSPRT across 1,000 replications of an experiment with Gaussian outcomes, N 4000, and five covariates.
23

---

H.1.3 Larger Minority Group
Fig. 2 in the main text summarizes CLASH's performance in an experiment where the minority group comprises 12.5% of the population. We now increase the size of the minority group, and present results in Fig. S6. With 25% minority group, CLASH improves stopping probability over the homogeneous baseline, though the gains are more modest than for 12.5% minority group. CLASH also performs slightly better than SUBTLE, though this difference is also reduced. All methods perform similarly with a 50% minority group.
O'iien-Flemme
ISPRT
MaXSPRT
Bayesian Estimatonn
ga
g n
6
201
0z
Harmtul Treatment Minorly
Oracle a
CLASH ss SUBTLE
Homageneous
(a) Minority Group Size:
O'Brien-Fleming
Bayesion
30
02
15 62 14
10 n2
64 65
08
Hammul Treatment Eieect on Minority Gruup
Homogeneous
(b) Minority Group Size: 50%
Figure S6: Performance of CLASH in simulation experiment with Gaussian outcomes, five covariates, and minority group that forms (a) 25% or (b) 50% all participants. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with 95% CI) at any interim checkpoint.
24

---

H.1.4
Comparison with SUBTLE
Fig. 3 in the main text compares CLASH's performance to that of SUBTLE in when the treatment benefits the majority group and harms (or has no effect on) the minority. We present an analogous comparison in the case when the treatment has no effect on the majority group in Fig. S7. In most cases, CLASH increases the stopping probability over SUBTLE baseline, especially the first two interim checkpoints. The only exception is the setting with 10 covariates and 12.5% minority group, when CLASH and SUBTLE perform similarly.
21 suamun
56
184
06
16
0002 05
00 05 85 2. 50 26 33
0002 0e LD
00 06
LD 00 36 20
Harmful Treshwent Enfection Minority Groupp
Harmful Treatment Teaamlitteeteeefffeeeet Effecce Minorty Group
(a) Minority Group Size: 12.5%
(b) Minority Group Size: 25%
Figure S7: CLASH's performance improvement over the SUBTLE baseline with Gaussian outcomes, when the treatment has no effect the majority group. We plot the difference in stopping probability between CLASH (with an mSPRT) and SUBTLE with 95% CIs, where the minority group forms (a) 12.5% (b) 25% of all participants.
25

---

H.1.5 Comparison with Homogeneous baseline
We now compare CLASH to the homogeneous baseline across wide range of simulation settings. Fig. S8 compares the two methods when the treatment benefits the majority, while Fig. S9 compares them when the treatment has no effect the majority, CLASH increases stopping probability in almost cases.
Checkpeat
Checkpont
Checppet
Chackpose
Checkpore
Checkpoar
0.6
000
06
00
8 00
20.05 05 D0.77 0E00.0 T6 "DS 04.00000
30.25 Dene BETD 20 DEREOE TO "2 DE Harmfu Trestment Effect Minority Group
Harmful Treatment Efflect on Minority Group
(a) Minority Group Size: 12.5%
(b) Minority Group Size: 25%
Figure S8: CLASH's performance improvement
the homogeneous baseline with Gaussian outcomes, when the treatment benefits the majority group. We plot the difference stopping probability between CLASH (with OF-adjusted z-test) and the homogeneous baseline with 95% Cls, where the minority group forms (a) 12.5% or (b) 25% of all participants.
Checkpeint
Checkpont:
67
50 05 10 00
0.
15 00 02 06
30 C2 64 06 05 00 02 C4 OE 10 02 DE 05 18
Harmful Treatment floct on Minariy Group
Harmfu Treatment Effect on Minority Group
(a) Minority Group Size: 12.5%
(b) Minority Group Size: 25%
Figure S9: CLASH's performance improvement over the homogeneous baseline with Gaussian outcomes, when the treatment has no effect on the majority group. plot the difference stopping probability between CLASH (with OF-adjusted z-test) and the homogeneous baseline with 95%
Cls, where the minority group forms (a) 12.5% or (b) 25% of all participants.
26

---

H.1.6 Small sample size
We evaluate CLASH's performance with much smaller sample sizes (N = 200, 400, 1000). We consider an experiment in which the treatment harms the minority group and weakly benefits the majority (i.e., 0o -0.1). There are five covariates and the minority group forms 12.5% of the population. Fig. $10 presents results for CLASH with an OF-adjusted z-test. CLASH outperforms the homogeneous baseline by a wide margin in experiments with moderately small samples (N=1000). While the gap between CLASH and the homogenous baseline decreases as sample size decreases, CLASH still outperforms the bascline with few as 400 participants. However, with very small samples (N=200), neither CLASH nor the homogeneous baseline is likely to stop the experiment.
N 200
400
N 1000
1.0
Stopping Probability
3.94
8.7
6.6-
833
6.34
82
8.1-
30 Harmful Treatment Effect Minority Group
Oracle
CLASH
Homogeneous
Figure S10: CLASH's performance in simulation experiments with small samples sizes (N=200, 400, 1000). We consider experiments with Gaussian outcomes, five covariates, a weakly benefitted majority group, and 12.5% harmed minority group. We simulate 1, 1,000 trials and plot the stopping probability (with 95% CT) at any interim checkpoint of CLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.7 Large number of covariates
We evaluate CLASH's performance with high-dimensional covariate sets (100 and 500 covariates). We consider an experiment with 4,000 participants which the treatment harms 12.5% minority group and weakly benefits the majority (i.c., 0 = 1..)). Fig. S11 presents results for CLASH with an OF-adjusted z-test. CLASH robust to increasing dimensionality to a point, outperforming the homogeneous baseline even with 100 covariates. The extreme case with 500 covariates is more challenging: here, investigators may need to perform feature selection before running CLASH.
100 covariates
500 covariates
1.6
8.9-
0.E-
0.3
8.K-
Stopping
0.5
0.4
0.3
8.2
8.1
Harmful Ereateentffecct Minority Group
Oracle
CLASH
Hamageneous
Figure $11: CLASH's performance in simulation experiments with high-dimensional covariate sets (100 and 500 covariates). We consider experiments with Gaussian outcomes, weakly benefitted majority group, and a 12.5% harmed minority group. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with 95% CI) at any interim checkpoint of CLASH (with an Od-adjusted 7-test), the homogeneous baseline, and the Oracle.
27

---

H.1.8 Multiple harmed groups
We evaluate CLASH's performance with multiple (> 2) groups in the population. We refer to the x-axis effect size values as z, and consider two settings:
a) Three groups of unequal size (group size and effect size, respectively, in parentheses): one weakly benefited (87.5%, -0.1), one strongly harmed (6.25%, σ), and one weakly harmed (6.25%, z/2).
b) Four equally sized groups (effect sizes in parentheses): strongly benefited (-x), weakly benefited (-z/2), weakly harmed (z/2), and strongly harmed (r).
Fig. S12 displays the results. Overall, CLASH performs well: it stops more frequently than the homogeneous baseline across a range of effect sizes and as often as the Oracle for larger effects.
Stapping Probabbilty
6.
Stopeing Probalailtt
E E.E. 3 S 2 Ei t G
8.7
60
02
14
06
03
10
02 04
05
08
20
Effect on Strongly Harmed Minority Group
Effect on Strongly Harmed Minority Group
Oracle
CLASH
Homogeneous
Oracle
CLASH
Homogeneous
(a) groups
(b) 4 groups
Figure S12: CLASH's performance in simulation experiments with multiple harmed groups. We consider experiments with Gaussian outcomes, five covariates, and (a) three groups (one weakly benefited majority group, one weakly harmed 6.25% minority group, minority group), or (b) four equally sized groups (one strongly benefited, one weakly benefited, one weakly harmed, one strongly harmed). We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with 95% CI) at any interim checkpoint of CLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
strongly harmed 6.25%
H.1.9 Stochastic group membership
In previous experiments, the mapping from covariates to groups was deterministic. We now evaluate CLASH's performance in situations where the covariates map stochastically to the benefited and harmed groups We construct 25% minority group deterministically from covariates as before, but randomly assign p% of this group to be harmed. The remainder of the minority group and the majority group are both weakly benefited with effect size f -0.1. We consider experiments with N = 4,000 and five covariates, and plot results in Fig. S13. CLASH outperforms the homogeneous bascline both when p = 0.5 and p = 0.75.
0
1.4
Harmfll Treatment E flet Minorit Group
CLASH
Homogeneous
Figure S13: CLASH's performance in simulation experiments with stochastic group membership. We consider experiments with Gaussian outcomes, five covariates, and weakly benefitted majority group. p% of participants from a 25% minority group are harmed by the treatment; the rest are weakly benefitted. We vary p between 0.5 and 0.75. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with 95% CI) any interim checkpoint of CLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
28

---

H.1.10 Smaller / larger outcome variance
We evaluate CLASH's performance as the variance in the outcomes increases. We consider an experiment with 4,000 participants and five covariates in which the treatment harms 12.5% minority group and weakly benefits the majority (0p -0.1). We vary Var(Y) between 0.5, 1, and (recall that VYr in previous experiments). Fig. S14 demonstrates that CLASH outperforms homogeneous baseline in all settings. CLASH and the Oracle both perform better with lower variance.
Variance =
Varlance =1
Variance =2
1.0
0.8
Probabiity
8.5-
0.7-
8.6
0.5-
0.44
0.3
0.2
0.4
0.6 133 ola
04 0.6 D8 010 0.2
06 08 12
Harmful Treatment Minority Group
Oracle
CLASH
Homogeneous
Figure S14: CLASH's performance in simulation experiments with smaller and larger variance in the observed outcomes. We consider experiments with Gaussian outcomes, five covariates, weakly benefitted majority group, and a harmed 12.5% minority group. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with 95% CI) any interim checkpoint of CLASH (with an OF-adjusted z-test), the homogeneous baseline, and the Oracle.
H.1.11 Unknown outcome variance
We now consider setting in which the variance in the outcomes unknown. The setup is exactly the same as in App. H.1.10 10, except that now we do not know Var(Y). Instead, we estimate the variance from the observed outcomes and use the plug-in version of the z-test. Fig. $15 demonstrates that while estimating the variance has a small effect on CLASH's performance, CLASH still drastically outperforms the homogeneous baseline.
Variance = 0.5
Varlance =1
Variance =2
+0-
0.6
0.7-
0.6
0.4
0.3
0.3
0.0
15.00
D. 00
06 16
Harmful Treatment E fffectton on Minority Group
Oracle
CLASH
Homogeneous
Figure $15: CLASH's performance in simulation experiments with unknown outcome variance. We consider experiments with Gaussian outcomes, five covariates, a weakly benefitted majority group, and harmed 12.5% minority group. We simulate 1,000 trials with 4,000 participants each and plot the stopping probability (with 95% CI) at any interim checkpoint of CLASH (with an OF-adjusted z-test with plug-in variance), the homogeneous baseline, and the Oracle.
29

---

H.2 TTE Outcomes
H.2.1 Setup
We consider clinical trial that measures time to positive event (e.g., remission), and adapt the simulation setup from [20]. The trial runs for 30 months and recruits 2, 000 participants, who accrue uniformly at random over the study's s first year. Some participants drop out of the trial: drop out time follows an exponential distribution with an annual hazard rate of 0.014. Group membership and covariates are generated as in the Gaussian case. The treatment effect is measured using the hazard ratio (HR), where R >> is beneficial and HR < 1 is harmful. Outcomes follow survival function with S(L) exp( -0. (1 D)t 0 ( G) - 0, DGi), with 01 E [0.05, 0.1] and 8, {0.1, 0.12). The treatment thus either benefits has non effect the majority group, and either harms or has no effect on the minority group.
We conduct checkpoints at 12, 18, and 24 months." CLASH's Stage uses survival causal forest [8] and 6 0.05. Stage uses an OF-adjusted Cox proportional hazards regression. Note that SPRT-based tests have not yet been adapted to the TTE setting; among other reasons, the statistical dependence between checkpoints (induced by observing additional data from the same participant) proves challenging for these tests. Crucially, SUBTLE cannot be used either; CLASH is thus the first heterogeneous stopping method applicable to this setting. We compute CLASH's stopping probability at each checkpoint across 1,000 replications, and compare it to the homogeneous baseline and oracle.
As with the Gaussian experiments, each replication uses its own random seed. One replication- including all combinations of effect size, minority group size, number of covariates and stopping tests-takes between 137 and 202 minutes run on single CPU (depending on the random seed). This yields a total compute time of approximately 3,000 hours for the TTE experiments. The simulations were run in parallel on an academic computing cluster with over 200 CPUs. Note that each replication used one CPU with 4GB of RAM.
Hirorty
1.0
0.8-
0.5-
0.3
02
3 0.
e
1.0
g
0.8
a
9.
0.3
02
0.0
06
0'a
1.0
Hazard Ratio Minority Group
Homageneous
Figure $16: Performance of CLASH in a simulation experiment with TTE outcomes. If the minority group is harmed (lower hazard ratio), CLASH (red) significantly increases the stopping probability over the homogeneous approach (blue). This improvement is most notable when then treatment benefits the majority group. We simulate 1,000 trials with 2,000 participants and five covariates and plot the stopping probability (with 95% CT) at any interim checkpoint.
We define checkpoints terms of time, TTE trials update observations for the same participants.
30

---

H.2.2 Results
CLASH (with an OF-adjusted Cox regression) significantly improves stopping probability over the homogeneous baseline if the treatment benefits the majority group and harms the minority (Fig. S16). For 25% minority group size, this performance improvement is most notable for large effect sizes (HR S 0.6), while for 50% minority group size, the improvement is most notable for medium effects (HR between 0.7 and 0.8). CLASH not only stops more often, but also faster, with higher stopping probability at earlier interim checkpoints (Fig. S17). These performance improvements reduced when the treatment has effect the majority group (Fig. S18). Note that do not consider harmed group sizes smaller than 25%, as no considered method was able to detect harmful effects with TTE outcomes such small populations.
Checlpeini P
Checppen #
Chelpenn P
Erectpott
Cleckpeer et
Clicpeen ei
Hazard Ratio Mineriy
Hazard Ratio Minerly Group
(a) Minority Group Size: 25%
(b) Minority Group Size:
Figure S17: CLASH's performance improvement with TTE outcomes when the treatment benefits the majority group. CLASH significantly increases the stopping probability across range of harmful effect sizes. CLASH not only stops more often, but also faster, boosting the stopping probability at earlier checkpoints. We plot the difference in stopping probability between CLASH and the homogeneous baseline (with 95% Cls), where the minority group forms (a) 25% or (b) 50% of all participants. The treatment effect is given by the hazard ratio (smaller HR indicates greater harm). For the larger group size, CLASH's improvement is most notable for medium effects, as the homogeneous baseline is also able to detect large effects. All performance boosts are robust an increase in the number covariates.
Checkpolnt #5
Checkpoit: 22
Checkpont d8
Checkpokc.
Checkpeir.cc #2
Checkpeir 43
08
Hazard Rablo for Minorty
Hazard Rato Minorty Group
Minority Group Size: 25%
(b) Minority Group Size; 50%
Figure S18: CLASH's performance improvement with TTE outcomes when the treatment has effect on the majority group. We plot the difference in stopping probability between CLASH and the homogeneous baseline (with 95% Cls), where the minority group forms (a) 25% or (b) 50% of all participants. The treatment effect is given by the hazard ratio (smaller HR indicates greater harm).
31

---

I Real-world Application
We now present supplementary tables and figures that were referenced in our discussion of the real-world application in Sec. 5. We first present regional treatment effects, estimated with the whole dataset (Tab. S3) and at CLASH's stopping time (Tab. S4). We then present results from our semi-synthetic evaluation of CLASH's performance by sample size. In Fig. 4 in the main text, we only sampled from Region (harmed group) and Regions 5-8 (unharmed group); this subset yielded harmed group that comprised 29% of the total population. We now increase the size of the harmed group by including data from Regions 2-4. Note that we considered region as harmed the treatment effect estimated with the whole dataset (Tab. S3) positive and significant; all other regions were considered unharmed. Further note that Fig. 4 in the main text only included Region (instead of one of Regions 2, 3, 4) since it was the most harmed region (Tab. S3). The results of this semi-synthetic evaluation (Fig. S20) are similar to those discussed in Sec.
Table S3: Regional Effect of Treatment Performance Metric (estimated at experiment's end with 500,000 observations). We present coefficients from separate univariate negative binomial regressions.
Treutment Effect Estimate
Std. Error
p-value
Region
0.385
0.014
0.000
Region 2
0.065
0.016
0.000
Region
0,107
0.034
0.002
Region
0.056
0.010
0.000
Region
0.006
0.015
0.684
Region
0,014
0.017
0.424
Region
-0.048
0.021
0.023
Region 8
-0,311
0.025
0.000
Table S4: Regional Effect of Treatment on Performance Metric (estimated at checkpoint with 40,000
observations, i.c., when CLASH stops the experiment). We present coefficients from separate univariate negative binomial regressions.
Treatment Effect Estimate
Std. Eror
p-value
Region
0.453
0.051
0.000
Region
0.051
0.000
Region
0.250
0.130
0.1544
Region
-0.146
0.036
0.000
Region
-0.115
0.065
0.078
Region
0.260
0.055
0.000
Region
-0.253
0.077
0.001
Region
-1.292
0.083
0.000
Table S5: Mean stopping time in our empirical application. We shuffle our real-world dataset 1 1,000 times and compute mean stopping time (with standard errors) for CLASH, the homogeneous baseline, and the Oracle.
Method
Mean Stopping Time (Std. Error)
Oracle
48,500 (431)
CLASH
57,200 (609)
Homogeneous
64,420 (848)
32

---

Table S6: ATE estimation in our empirical application if the experiment is only stopped for the harmed group. We assume that the experiment stopped Region CLASH stopping time of 40,000 participants and continued in all other regions. We consider the ATE without stopping (i.e., after continuing to collect data from all regions) as the ground truth. The naive ATE estimate (i.c., an unweighted estimate using all collected data) is biased and underestimates the true ATE, Using the IPW approach detailed in App. corrects this bias.
Estimate (Std. Error)
ATE estimate without stopping Naive ATE estimate with stopping IPW ATE estimate with stopping
0.11 (0.006)
0.03 (0.006)
0.10 (0.006)
1000
Distribution of CLASH weights
0.75
0.50
0.25
0.00
Region Region 2 Region Region Region 5 Regian Region Region B
Figure S19: Distribution of CLtsH-estimated weights by region (at checkpoint with 40,000 observa- tions, i.e., when CLASH stops the experiment). CLASH identifies that the treatment has strong harmful effect in Regions I and 2 and assigns these observations unit weight.
33

---

Probeliiity of Stopping
0.6
0.5
probabiing
0.6
0.0
0.3-
0.3
0.2
0.1
0.0
0.0
Experiment (N,)
Size Experiment (N
Oracle
CLASH
Homogeneous
Oracle
CLASH Homogeneous
(a) Region
(b) Regions 1-2
1.0
Probabiiitt of Suappingg Early
0.6-
Probabing
0.3
83
0.2
02
0.1-
0.0-
0.0
20K
30K
40K
60c
70k
63k
50K
100C
10c
20k
40k
50k
60C
90k
100K"
Size of Experiment (N,)
Size
Experment
Oracle
CLASH
Homageneous
Hamageneous
(c) Regions 1-3
(d) Regions 1-4
Figure S20: CLASH's performance by sample size with real data from A/B test. We plot stopping probability across 1,000 random samples with 95% Cls. We define unharmed group as Regions 5-8. We define the harmed group as (a) Region (29% harmed group size), (b) Regions 1-2 (41 % harmed group size), (c) Regions 1-3 (43% harmed group size), or (d) Regions 1-4 (60% harmed group
size). In cases (a) (c), CLASH significantly increases stopping probability over the homogencous bascline for all N, 2 50,000, and achieves ocar- level performance with N, = 100,000 (20% of the overall experiment's size). Case (d) reflects situation in which majority group is harmed, which not our method's 's primary focus. However, CLASH still outperforms the homogeneous baseline by N, 100,000.
34


Reviews:
[{"review_id": "ltuOx2C3WR", "reviewer": "Reviewer_wJ1C", "summary": "A two-stage approach CLASH was proposed to determine the early stopping time of a randomized experiment when the treatment is harmful to a subset of the population. The indicators of the harmed groups are estimated by causal machine learning methods in stage 1, and the early stopping time is determined using the weighted test statistic in stage 2. Theoretical properties of the proposed method were established. Simulation studies were conducted to show the existing homogenous method's failure and CLASH's success. Finally, an illustration of the proposed methods was presented by analyzing real data from a digital experiment. Overall, the proposed methodology is useful, and the paper is well-written.", "questions": "See Weaknesses.", "limitations": "The author provided a paragraph to discuss the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The method is solid, clearly described, and useful in different areas. With the accommodation of both Gaussian and time-to-event experimental outcomes and considering the limited prior work on this specific area, the proposed method can make a good impact.  The results of the simulation studies and real data analysis are convincing and support their conclusions.", "weaknesses": "The simulation settings seem relatively simple and difficult to interpret: only several binary covariates were included, and there was only 1 harmed group. It would be interesting to see some simulation results under settings closer to the real data analysis (multi-level categorical covariates and multiple harmed groups). Including several sentences to interpret the simulation settings would also be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and positive feedback: we’re glad you found our work useful and impactful. We appreciate your constructive comments, which we address below and in our revised paper.  \n\n**R1: _The simulation settings seem relatively simple and difficult to interpret: only several binary covariates were included, and there was only 1 harmed group. It would be interesting to see some simulation results under settings closer to the real data analysis (multi-level categorical covariates and multiple harmed groups). Including several sentences to interpret the simulation settings would also be helpful._**\n\nIn our global response, we summarize results from additional simulation experiments that evaluate CLASH in settings closer to real-data analysis, per your suggestions. Specifically, Fig. 1 presents results with multiple harmed groups, while Fig 2e presents results with 100 and 500 binary covariates (similar to many multi-level categorical covariates). We find that CLASH outperforms the homogeneous baseline in these more realistic simulation settings, and hope that this finding encourages use of CLASH in real-world experiments. We include these new results, as well as more detail on interpreting these simulation settings, in Section 4 of the revised paper. \n"}, {"review_id": "pZUJMWUSyJ", "reviewer": "Reviewer_af9S", "summary": "This paper proposes a method to decide when to stop a clinical trial or an A/B study in cases where the treatment/intervention only causes harm to a minority group of participants, in which traditional methods can fail to detect the need for stopping an experiment. The paper includes a thorough theoretical analysis of the method, which suggests that it has desirable properties (stopping when a treatment is harmful and not stopping when it is not harmful) for large enough n. It also includes extensive simulations demonstrating that the method outperforms established methods in most cases and performs well in a real-world application.", "questions": "First of all, I want to say that this paper was a pleasure to read. I was impressed by the thoroughness and extend of the analyses. I only have a few suggestions. I hope you will find them helpful and constructive.\n\n**Major points**\n- I would like to see how the method holds up for much smaller sample sizes in simulations and on the empirical data, since there are many clinical trials with sample sizes around 50, 100 or 200 participants. Even if the method does not work well for this, it would be an important information to have so that practitioners do not apply this method when it’s not appropriate.\n- On p. 3, you write “For example, consider a situation in which there are two equally sized groups with equal but opposite treatment effects, that is, p(G = 0) = p(G = 1) and τ (0) = −τ (1). The ATE is zero, and so any stopping test with H0 : ATE ≤ 0 is designed to continue to completion at least (1 − α)% of the time.” However, later you focus on Gaussian outcomes. Can your method handle the first scenario as well (i.e., bimodal distributions)? Or could it be extended to Gaussian mixtures for example?\n\n- In the simulations shown in the supplement (Figure S6), there are certain scenarios for which the homogenous baseline outperforms CLASH (e.g., panel A: Bayesian Estimation MaxSPRIT, no treatment effect on the majority and small harmful effects or panel B: MaxSprit for small effects). Can you speculate on why this is the case? I do not think it is a problem if your method is not the best in every scenario, but it is important to outline and understand the conditions under which it is outperformed by other methods.\n\n**Minor points**\n- How would this method be used to decide for whom a trial should be stopped early in practice? Would this be based on the probability of being harmed, does this require a threshold? Perhaps you can elaborate more on this. \n- Out of curiosity, could this method be extended to multiple groups that show different harm (e.g., a strongly and a weakly harmed group)?\n", "limitations": "The authors adequately discuss the limitations of their work. Depending on the performance for much smaller n in simulations it might be worthwhile expanded on the required sample size as well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is well written and easy to follow.\n- The research question is interesting and important also from a fair AI perspective as this research protects minority groups.\n- The paper includes a very extensive analysis of the problem, including theory, simulations and an empirical application.\n", "weaknesses": "- The simulations could include a wider range of sample sizes, i.e. sample sizes of 50-200 participants which are common in clinical trials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your detailed review and positive feedback: we’re glad you enjoyed reading our paper and appreciate your encouragement. We address each of your constructive suggestions below, which we believe have further strengthened the paper. \n\n**R2: _The simulations could include a wider range of sample sizes, i.e. sample sizes of 50-200 participants which are common in clinical trials…I would like to see how the method holds up for much smaller sample sizes in simulations and on the empirical data, since there are many clinical trials with sample sizes around 50, 100 or 200 participants. Even if the method does not work well for this, it would be important information to have so that practitioners do not apply this method when it’s not appropriate._**\n\nThank you for this important suggestion. Fig 2d in our global response considers an experiment with smaller sample sizes (N=200, 400, 1000). Overall, we find that CLASH can be effective in small samples: it outperforms the homogeneous baseline with as few as 400 participants, and with 1000 participants, the performance gap is wide. However, N=200 is a challenging scenario: neither CLASH nor the homogeneous baseline is able to stop experiments with such few participants. We have included this setting as a limitation in our revised paper. \n\n**R2: _On p. 3, you write “For example, consider a situation in which there are two equally sized groups with equal but opposite treatment effects, that is, p(G = 0) = p(G = 1) and τ (0) = −τ (1). The ATE is zero, and so any stopping test with H0 : ATE ≤ 0 is designed to continue to completion at least (1 − α)% of the time.” However, later you focus on Gaussian outcomes. Can your method handle the first scenario as well (i.e., bimodal distributions)? Or could it be extended to Gaussian mixtures for example?_**\n\nApologies for the confusion on this point. Our simulation experiments do in fact consider this setting: the outcomes are Gaussian within each group (i.e., majority and minority), not the population as a whole. Thus, viewed over the whole population, the outcomes are generated from a mixture of two Gaussians. We have clarified this point in the revised paper.\n\n**R2: _In the simulations shown in the supplement (Figure S6), there are certain scenarios for which the homogenous baseline outperforms CLASH (e.g., panel A: Bayesian Estimation MaxSPRIT, no treatment effect on the majority and small harmful effects or panel B: MaxSprit for small effects). Can you speculate on why this is the case? I do not think it is a problem if your method is not the best in every scenario, but it is important to outline and understand the conditions under which it is outperformed by other methods._**\n\nThank you for giving us an opportunity to address this point. In general, CLASH performs better when it is easy to differentiate between the harmed and unharmed groups from the covariates and outcomes. The more similar the effects on the minority and majority groups, the harder this task is; CLASH thus performs best when |majority effect - minority effect| is large. The homogeneous baseline is the exact opposite: it performs better when the whole population ATE is more similar to the effect on the minority group. Thus, the case in which the treatment has no effect on the majority but a small harmful effect on a large minority group (25% or 50%) is a near-ideal situation for the homogeneous baseline, but a more difficult situation for CLASH. However, it is worth noting that the difference in stopping probability between CLASH and the homogeneous baseline is relatively small, even in this difficult situation. We have added this discussion to the revised paper to provide more insight for practitioners.\n\n**R2: _How would this method be used to decide for whom a trial should be stopped early in practice? Would this be based on the probability of being harmed, does this require a threshold? Perhaps you can elaborate more on this._**\n\nThank you for raising this important point. We have addressed this in point (2) of our global response. In short, the decision will heavily depend on domain specifics, but CLASH weights can provide heuristics that can help inform practitioner decisions. \n\n**R2: _Out of curiosity, could this method be extended to multiple groups that show different harm (e.g., a strongly and a weakly harmed group)?_**\n\nYes, CLASH can be leveraged in this scenario: see Fig 1 in our global response for a demonstration.\n"}, {"review_id": "wetHIeq0Ed", "reviewer": "Reviewer_K2p1", "summary": "The authors propose CLASH, a method for early stopping in RCTs and A/B tests on heterogeneous populations. They some theoretical results that show that CLASH works, and provide simulations and one real experiment.", "questions": "- In the simulation you have \"X maps deterministically to G\" -> is this not unrealistically simple? What happens if you use a stochastic function with varying degrees of noise?\n- Similarly, \"recruiting one treated and one untreated participant at each step\" is unrealistic. Why not try to mimic real recruitment procedures and have batch recruitment? Does this affect your results at all?\n- In your simulation \"Y is normally distributed\" with a fixed standard deviation of 1.0. What happens if the standard deviation is larger or smaller? Is it important to know this standard deviation in practice?\n- You write \"All performance increases are robust to an increase in the number of covariates\" but also \"CLASH works better with a relatively small number of covariates\": these two statements conflict, and I did not see any experiments that demonstrate the latter. It's important to add experiments that quantify how your models perform (or don't) in higher dimensions. What happens when the dimensionality is 100 and 1000 in the simulation?\n- \"We only sample from Region and Regions 5-8; this gives us one harmed group (Region 1) that comprises 28% of the total population\" - why do this? Why not sample uniformly at random from the entire dataset?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clear writing\n- Good motivation\n- Excellent exposition of theoretical results for readers who may not be able to fully understand proof details\n- Convincing (albeit limited) experimental evidence", "weaknesses": "The biggest issue with this paper has to do with the experiments. \n\nEDIT: I have raised my score to a 7 after the authors answered my questions.\n\n- Unrealistic settings in the simulation - see the questions section.\n- Your real-world data experiment is nice, but it's a single experiment with a single outcome - you could be doing as well as Oracle purely by chance. You could run it multiple times by randomizing the order of the data and reporting a distribution of stopping times against oracle, SUBTLE, and homogeneous. \n- \"Note that stopping the experiment just in one region would affect statistical inference at the end of the experiment, as the treatment would no longer be randomly assigned across regions. Practitioners can use covariate adjustment, inverse probability weighting, or adaptive sampling methods [13] to adjust for this selection.\" I would recommend actually performing this calculation and reporting the outcome.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review: we greatly appreciate your insightful feedback. We address each of your constructive suggestions below, which we believe have further strengthened the paper.  \n\n**R3: _The biggest issue with this paper has to do with the experiments. I'll happily raise my score to a 7 once additional experimental results are included…_**\n\nOur global response presents results from additional simulation experiments, many of which directly address your comments below. We include these results in the appendix of the revised paper.\n\n**R3: _Your real-world data experiment is nice…you could run it multiple times by randomizing the order…_**\n\nThank you for raising this important point. We have included your suggested analysis in the revised paper. We find that CLASH stops the experiment at the same interim checkpoint as the Oracle in 62.6% of shuffled datasets. The mean (std. error) stopping times for each method across 1,000 shuffles is below.\n\nCLASH: 57,200 (609)\n\nHomogeneous: 64,420 (848)\n\nOracle: 48,500 (431)\n\n**R3: _‘Stopping the experiment just in one region would affect statistical inference’... I would recommend...performing this calculation…_**\n\nThank you for this suggestion. We report results from our empirical application below and in the revised paper. We assume practitioners choose to stop the experiment in the region with the largest harmful effect (Region 1). This choice leads to bias in the naive estimate of the full population ATE; however, inverse propensity weighting (IPW) corrects the bias and successfully recovers the ATE.\n\nATE estimate without stopping: 0.11 (0.006)\n\nNaive ATE estimate with stopping: 0.03 (0.006)\n\nEstimate using IPW: 0.10 (0.006)\n\n**R3: _X maps deterministically to G -> is this not unrealistically simple? What happens if you use a stochastic function...?_**\n\nYour comment is well-taken, especially for clinical trials. To address your question, Fig 2a in the global response demonstrates that CLASH still performs well even with stochastically determined groups. That said, we would like to note that deterministic mappings from covariates to harmed groups do occur in real-word settings. For example, a new product feature evaluated in an A/B test may increase system crashes for only certain device types (e.g., Android devices with a certain chip). In such cases, the harmed groups can be determined entirely from the covariates. \n\n**R3: _'recruiting one treated and one untreated participant at each step' is unrealistic. Why not…have batch recruitment?_**\n\nThank you for giving us the opportunity to clarify. This assumption is actually not necessary for CLASH, but rather for certain stopping tests that CLASH can be used with. For example, the mSPRT (Johari et al., 2017) requires this assumption, as it considers an “observation” at each time step to be the difference in outcomes between a treated and untreated unit. Note that this is not as strong an assumption as it may initially seem: if the data is i.i.d. (as we assume), then at any interim checkpoint, the data collected thus far can be re-ordered to ensure that there is a treated and untreated observation at every step. That said, it does complicate estimation in cases where the treatment-control split is not 50-50.\n\nHowever, we emphasize that CLASH does not require this assumption if used with non-SPRT techniques. For example, CLASH with the O’Brien-Fleming test can be used with batch recruitment with imbalanced treatment and control groups. This does not affect our results: for example, Figs 2b and 2c in our global response (which focus primarily on outcome variance) do not assume one treated and untreated participant per time step. We demonstrate that CLASH is still able to perform well in these settings. \n\n**R3: _What happens if the standard deviation is larger or smaller? Is it important to know this standard deviation in practice?_**\n\nFigs 2b and 2c in the global response consider these scenarios. We find that CLASH and the Oracle are both affected by increasing variance (Fig 2b). Notably, CLASH outperforms the homogenous baseline for all settings. Needing to estimate the variance (instead of knowing it a priori) does have an effect on CLASH’s performance (Fig 2c); however, CLASH still outperforms the homogeneous baseline across all considered variances. \n\n**R3: _What happens when the dimensionality is 100 and 1000 in the simulation?_**\n\nFig 2e in the global response considers a high-dimensional setting with 100 and 500 covariates. We find that CLASH is still able to perform well with 100 covariates, outperforming the homogeneous baseline for medium and large effect sizes. With 500 covariates, CLASH only stops the experiment slightly more often than the homogeneous baseline for large effects. This result thus illustrates CLASH’s limitations in dealing with very high-dimensional covariate sets. We note, however, that 500 covariates is a fairly extreme setting for an experiment with 4,000 participants. In practice, even if 500+ covariates are available, practitioners would be able to use their domain expertise or statistical methods (e.g., LASSO) to specify a subset of covariates on which to assess harm. We have updated our limitations section to reflect this discussion, and recommend feature selection before running CLASH in very high-dimensional settings. \n\n**R3: _'We only sample from Region 1'... Why not sample uniformly at random…?_**\n\nThis is a great question – in fact, we do present results from uniform sampling in Fig S14 in App H of the submitted paper. CLASH is able to outperform the homogeneous approach in this setting, though only for larger experiments (80k+ participants). However, CLASH’s main focus is on experiments in which a minority group of participants is harmed. If we were to uniformly sample all regions, the harmed group (Regions 1-4) would form the majority (60% of all participants); we thus only sample from Region 1 to illustrate our main contribution."}, {"review_id": "gRulGOXtuE", "reviewer": "Reviewer_SD94", "summary": "The authors propose an approach to early stop clinical trials in order to prevent subgroup level harms. Their approach involves first estimation of sequential estimation of a an individualized treatment effect using machine learning methods followed by reweighting the test statistic at each iteration with the estimated mean and standard deviation of the CATE. \n\nFurthermore their choices are such that they do not need to make apriori assumptions on what groups represent the minority subgroups. \n\nThe authors present results around optimality of the metrics using the estimated CATE and perform extensive real world and synthetic experiments to demonstrate the effectiveness of their approach. ", "questions": "* The cdf for chosen weights, $w$ corresponds to a normal distribution. I think this is because the estimated weights tend to a normal distribution by central limit theorem, but I was unable to find a result for this in the paper can the authors point to this. \n\n* Can the authors address the second weakness pointed out in more details. Specifically does the current setup allow a practioner to specify something along the lines of a \"minimum\" group size to prevent the model from raising false alarms with trivial subgroups?\n\n* Finally, the current setup it seems does not allow for anyway of specifying how the covariates across different subgroups are related. For a subgroup to be actionable there should be similarities in there covariates. Is there a way to enforce or obtain such similarities from the experiment.\n \nOverall I am willing to readjust my scores favorably based on answers to my questions as well as discussions and deliberations during the author rebuttal phase.", "limitations": "The paper does raise an interesting concern over whether current trials should be stopped for the entire population if a single subgroup is found to be harmed from the intervention. As the authors point out, this is a question of medical ethics and is largely beyond the scope of this manuscript, and hence I am inclined to not consider it as a \"limitation\". I think however that perhaps some examples of where such decisions can lead to different outcomes should be included in the manuscript.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem of early stopping of a clinical trial to prevent aggregate and subgroup level harms is an important one. The contribution is timely and relevant. \n\n* The demonstration of the proposed method on the time-to-event (survival) setting is welcome since outcomes in most real world clinical trials are censored time-to-events. \n\n* The extensive theoretical insights and real world and synthetic experiments are thoughful and welcome.", "weaknesses": "* The paper requires estimates of the uncertainity in CATE at an *individual* level. CATE is itself hard to estimate, and its uncertainity is even harder in practice to estimate. The authors propose to use \"bootstrap\" to compute this quantity. This seems practically impossible especially with high dimensional covariates.\n\n*  While the paper is motivated strongly there is a fundamental question that remains to be addressed: The current method is such that it does not require apriori assumptions on which covariates specify the subgroups. It is unclear as to how in practice the group would be specified implicitly by the model. For instance, in the case of a large RCT there would always exist trivially small subgroups that are harmed. However these subgroups would not be generalizable and such results would not transport. Ideally this should be reflected in the uncertainity estimates upto generalization error. However this maybe violated in small sample size regimes.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review: we appreciate your positive feedback and constructive suggestions. We discuss each of your comments in detail below, and incorporate our responses in the revised paper. \n\n**R4: _The authors propose to use \"bootstrap\" to compute this quantity. This seems practically impossible especially with high dimensional covariates._**\n\nThank you for allowing us to clarify. You are correct: CLASH requires estimating uncertainty in individual CATE estimates. However, the bootstrap is merely one way to obtain such uncertainty estimates. For example, a causal forest estimates these uncertainties in the process of fitting, and thus requires little additional computation. We recommend using such methods with CLASH wherever possible. Bootstrapping is a last-resort in cases where there are strong domain-specific reasons to use a certain CATE estimation method, and that method is unable to estimate the required uncertainties on its own. We have added this suggestion to the methods section of our revised paper.\n\n**R4: _While the paper is motivated strongly there is a fundamental question that remains to be addressed…it is unclear as to how in practice the group would be specified implicitly by the model. For instance, in the case of a large RCT there would always exist trivially small subgroups that are harmed. However these subgroups would not be generalizable and such results would not transport._**\n\nThanks for raising this important point. As detailed in Part 2 of our global response, when CLASH indicates that an experiment should be stopped, practitioners can identify the specific harmed groups either by analyzing the distribution of the estimated CLASH weights or by using existing subgroup identification methods (see global response for more details). We also suggest ways to ensure that the identified harm subgroups are not trivially sized (e.g., limiting depth for tree-based methods). In general, domain expertise must play an important role in determining what is and what is not a meaningful group on which to stop. \n\nRegarding inaccurate uncertainty estimates in small samples: in Part 1 of the global response, we find that CLASH rarely stops experiments with very low N (Fig 2d). Thus, CLASH displays the opposite problem with small samples: it does not stop experiments due to trivially-sized harmed groups, but rather is unable to detect harmful effects with low N (note that the homogenous baseline is also unable to detect such effects). In general, we advise using CLASH with caution in experiments with small samples; we discuss this further in our limitations section.\n\n**R4: _The cdf for chosen weights corresponds to a normal distribution. I think this is because the estimated weights tend to a normal distribution by central limit theorem, but I was unable to find a result for this in the paper can the authors point to this._**\n\nThank you for giving us the opportunity to clarify. The normal CDF is used primarily because it leads to fast convergence of the CLASH weights to the optimal weights (see Thm 3.2), not because the weights have an asymptotically normal distribution. It may be possible to use other CDFs and achieve similar results; however, the normal CDF proved to be an effective choice, as it yields provably fast weight convergence. We emphasize that CLASH does not require any central limit theorem-like assumptions on either the weights or the estimated CATEs.\n\n**R4: _Can the authors address the second weakness pointed out in more details. Specifically does the current setup allow a practioner to specify something along the lines of a \"minimum\" group size to prevent the model from raising false alarms with trivial subgroups?_**\n\nPer Part 2 of our global response, we recommend a tree-based heuristic that practitioners can use to identify the harmed groups. If this heuristic implies that only a trivially sized group is harmed, practitioners can ignore CLASH’s recommendation and continue the experiment. However, this decision depends heavily on the ethical and financial considerations of the practitioners.\n\n**R4: _...For a subgroup to be actionable there should be similarities in there covariates. Is there a way to enforce or obtain such similarities from the experiment._**\n\nThank you for raising this concern. We agree that to stop an experiment only on one group, the group should have similar values for a few key covariates. This relates closely to our discussion above on identifying the harmed group (further detailed in point 2 of our global response). Practitioners can identify such harmed groups in an actionable way by using either our tree-based heuristic or existing subgroup identification techniques (e.g. [1]). Limiting the depth of the tree-based heuristic offers an easy way to ensure the identified group is actionable and of non-trivial size. For example, by limiting the tree-depth to 2, investigators can identify the two covariates that most drive harm and get a well-defined group on which to stop the experiment (if this is what they choose to do). Subgroup identification techniques provide analogous ways to get actionable groups (e.g. limiting tree-depth, variable selection, etc.).\n\n[1] Zhang, et al, DOI: 10.21037/atm.2018.03.07\n\n**R4: _Limitation: …perhaps some examples of where such decisions can lead to different outcomes should be included in the manuscript._**\n\nWe address this limitation in further detail in part (2) of our global response, including examples from both clinical and technology domains of whether / when to stop on a subgroup; we will also provide these examples in the revised paper. \n\n**R4: _Overall I am willing to readjust my scores favorably based on answers to my questions as well as discussions and deliberations during the author rebuttal phase._**\n\nThank you for your detailed feedback. We hope our response has helped answer your comments, and look forward to further discussion this week.\n"}, {"review_id": "EibzaPWSyZ", "reviewer": "Reviewer_R8bL", "summary": "The paper focuses on stopping tests for harm in clinical trials or A/B testing where heterogeneous treatment effect is involved. The proposed method contains two phases: First, the population harmed by the trials is identified via conditional treatment effect estimation. Second, weighted versions of widely-used test statistics are computed to determine if the trial needs to be stopped early. Theoretical analyses are carried out to show that the proposed method meets the desired requirements: producing high probability when a subgroup is harmed while limiting the unnecessary stopping. Experiments using simulation and real-world data show improvement over homogeneous stopping tests and the existing heterogeneous ones.", "questions": "- In the lower part of Fig. 2, when the treatment has no effect on the majority, why CLASH has lower probabilities of stopping early than the homogeneous approach when the harmful treatment effect on the minority group is below 0.4?", "limitations": "The limitations of the proposed method are discussed in the last section. It would be great if the authors could add a few sentences about how they plan to address the limitations in future work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper is well-written and easy to follow. The research problem is clearly defined and the motivation for the proposed method is nicely presented.\n+ Theoretical analyses show that the proposed method could achieve the desired properties. The convergence property (Thm. 3.2.) is particularly important and well-explained.", "weaknesses": "- In lines 191 to 193, $(x_i, y_i)$ is excluded from training set when estimating $\\tau(x_i)$. Does it mean that the CATE estimation model needs to be trained for $n$ times during stage 1? Are there any comparisons of the running time between baseline models, say SUBTLE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review: we appreciate your positive feedback and constructive suggestions. We address each of your comments in further detail below. \n\n**R5: _In lines 191 to 193, $(x_i,y_i)$ is excluded from training set when estimating $\\tau(x_i)$ Does it mean that the CATE estimation model needs to be trained for n times during stage 1? Are there any comparisons of the running time between baseline models, say SUBTLE?_**\n\nOur apologies for not being more clear. While it is possible to use leave-one out cross-validation---in which case the model would need to be trained n times---we recommend using k-fold cross-validation or progressive cross-validation instead. In this case, the model would only need to be trained k times during stage 1. Our simulation experiments and empirical application use 5-fold cross-validation; we will make this more clear in the revised manuscript. In our empirical application, at an interim checkpoint with 40,000 participants (i.e., when CLASH indicates that the experiment should be stopped), all four methods take under a minute to run, though CLASH is the slowest (clocking in at 36 seconds).\n\n**R5: _In the lower part of Fig. 2, when the treatment has no effect on the majority, why CLASH has lower probabilities of stopping early than the homogeneous approach when the harmful treatment effect on the minority group is below 0.4?_**\n\nThank you for raising this important point. Overall, the situation in which the majority group is unaffected and the minority group is only slightly harmed reflects both an easy case for the homogeneous baseline and a difficult case for CLASH. CLASH performs better when it is easy to differentiate the harmed and unharmed groups based on the covariates and outcomes. The more similar the effects on the minority and majority groups, the harder this task is. Meanwhile, the homogeneous baseline is the exact opposite: it performs better when the whole population ATE is more similar to the effect on the minority group. However, we emphasize that even in this case, the difference between the two methods is small and only exists for the Bayesian estimation-based stopping test."}, {"review_id": "vxFv6OaEQU", "reviewer": "Reviewer_tX2R", "summary": "This work proposes a method for adapting stopping tests of randomized experiments in heterogeneous populations. Specifically, the authors motivate the problem, namely why heterogeneous treatment effects lead to late stopping of randomized experiments, for instance when a minority group is harmed. They then propose a two stage method which first predicts a weighting of the original test statistic components used in stopping decisions, then and then uses these statistics to make the stopping decision. The methodological contribution is well-motivated and supported by theoretical results which analyse the convergence behavior of these weights, and the probability of stopping under the assumption of knowing the group membership.", "questions": "* The problem setup of CLASH stops the entire experiment if a stopping decision has been made. However, in clinical trials for example, it may not be ethical to stop the experiment for a majority group which benefits from the treatment. Could CLASH be adapted to an online setting where the experiment is only stopped for the harmed subgroup? It would be interesting if the authors could comment on this. ", "limitations": "* Prop. 3.1 assumes the group membership (and CATE) is known, which is never the case in practice. The authors state this, yet it is unclear how one would efficiently infer group membership in large-scale scenarios in practice, and whether the proposed solution in Stage 1 of the algorithm would work. It is consequently unclear how well the proposed tests would generalise if group membership is not known, or how the asymptotic behavior analysed in this Proposition would change in this case. It would be helpful if the authors could comment on this.\n* The experimental settings are all limited to two groups. In many real-world settings such as clinical trials, we would expect more than two groups with somewhat homogeneous treatment effect. How this method would perform in such cases is unclear, and neither discussed in the paper.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* There is a lack of machine learning methods that addresses the heterogeneous early stopping problem, and this paper provides a possible, first solution while making few assumptions. This renders the work an original, well-motivated contribution.\n* The paper is very clear and well written. In particular, the links between each of the sections are very clear.\n* The experimental results, including the simulated scenarios, are convincing and interesting. For instance Figure 2 makes clear why the proposed approach is advantageous over homogeneous stopping tests. \n", "weaknesses": "* The task is somewhat niche. It is furthermore unclear to what degree the stopping task in randomized experiments could be reformulated as a similar task in another domain, i.e. to what degree this or similar problems have been solved in other contexts.\n* Overall, the work makes various idealised assumptions in both the theoretical results, and in the (synthetic) experiments considered. It is an interesting proof of concept, but there is a lot of work which would need to be done to make this method applicable in practical settings, for instance real-world clinical trials which this method is motivated by, but does not evaluate on. Two further points to note is the lack of performance on high-dimensional data that the authors themselves notes, which is present in some clnical trials. Second, the method crucially relies on treatment effect estimation methods in Stage 1, which themselves are far from being widely applicable in practice. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review: we appreciate your positive feedback and constructive suggestions. We address each of your comments in further detail below. \n\n**R6: _Overall, the work makes various idealised assumptions in both the theoretical results, and in the (synthetic) experiments considered. It is an interesting proof of concept, but there is a lot of work which would need to be done to make this method applicable in practical settings, for instance real-world clinical trials which this method is motivated by, but does not evaluate on. Two further points to note is the lack of performance on high-dimensional data that the authors themselves notes, which is present in some clinical trials. Second, the method crucially relies on treatment effect estimation methods in Stage 1, which themselves are far from being widely applicable in practice._**\n\nThank you for raising this important point. Our global response considers several additional simulation settings, including on high-dimensional data (Fig 2e in the global response). We find that CLASH is able to perform well even in experiments with 100 covariates. Overall, we believe the additional simulations indicate the CLASH can be effective in situations closer to real world clinical trials and A/B tests. \n\nFurther, we emphasize that CLASH does not require using machine learning-based methods for causal estimation. Practitioners can use much simpler techniques–including linear regression–to infer heterogeneous treatment effects in stage 1. CLASH is agnostic to the specific method used: practitioners can use the method with which they are most comfortable, as long as it yields reasonably accurate estimates of the effect heterogeneity. \n\n**R6: _The problem setup of CLASH stops the entire experiment if a stopping decision has been made. However, in clinical trials for example, it may not be ethical to stop the experiment for a majority group which benefits from the treatment. Could CLASH be adapted to an online setting where the experiment is only stopped for the harmed subgroup? It would be interesting if the authors could comment on this._**\n\nThis is an excellent point: we discuss this in detail in point (2) of our global response and have updated our manuscript accordingly to address this decision-making process. In short, yes – CLASH can be used to inform early stopping on only the harmed subgroup rather than the entire trial population.\n\n**R6: _Prop. 3.1 assumes the group membership (and CATE) is known, which is never the case in practice. The authors state this, yet it is unclear how one would efficiently infer group membership in large-scale scenarios in practice, and whether the proposed solution in Stage 1 of the algorithm would work. It is consequently unclear how well the proposed tests would generalise if group membership is not known, or how the asymptotic behavior analysed in this Proposition would change in this case. It would be helpful if the authors could comment on this._**\n\nThank you for giving us the chance to clarify: group membership knowledge is not needed when using CLASH. You are correct to point out that the test in Prop 3.1 could never be used in practice, as it requires prior knowledge of the groups and treatment effects. However, it is an important result because it describes the statistical power of the test that CLASH converges to in large samples. Thm 3.2 establishes that in large samples, CLASH converges to the test described in Prop 3.1. Prop 3.1 indicates that this test has power-1 in large samples; thus CLASH must also have power-1 in large samples. We emphasize that CLASH does not require knowledge of group membership to operate: it infers this in stage 1 using the provided covariates and observed outcomes. Thus, CLASH is able to obtain the optimal power of the test described in Prop 3.1 despite not knowing group membership a priori. Our simulation experiments and empirical application demonstrate that CLASH efficiently infers group membership in practice.\n\n**R6: _The experimental settings are all limited to two groups. In many real-world settings such as clinical trials, we would expect more than two groups with somewhat homogeneous treatment effect. How this method would perform in such cases is unclear, and neither discussed in the paper._**\n\nThank you for raising this important concern. Figure 1 in our global response considers situations with more than two groups, and demonstrates that CLASH can be effective in such settings. "}]

Author rebuttal:
We thank the reviewers for their positive comments and constructive feedback. In this global response, we focus on two themes raised by multiple reviewers: (1) additional experiments, and (2) the decision to stop only on the harmed group. We separately provide responses to individual reviewers.

### (1) Additional Experiments
The attached pdf contains figures showing CLASH’s performance in reviewer-suggested experiments. All figures use Gaussian outcomes and the O’Brien-Fleming stopping test; more experiments will be added to the paper supplement.

**Figure 1**: CLASH’s performance for trial populations with >2 groups. CLASH performs well: it stops more frequently than the homogeneous baseline across a range of effect sizes and as often as the Oracle for larger effects. We consider two settings:

a. Three groups of unequal size (group size and effect size, respectively, in parentheses): one weakly benefitted (87.5%,  -0.1), one strongly harmed (6.25%, x-axis), and one weakly harmed (6.25%, x/2).

b. Four equally sized groups (effect sizes in parentheses): strongly benefitted (-x), weakly benefitted (-x/2), weakly harmed (x/2), and strongly harmed (x).

**Figure 2**: Specific reviewer-suggested experiment settings. In all figures, the treatment harms the minority group and weakly benefits the majority (effect size: -0.1). Unless specified, there are 5 covariates and minority group size is 12.5%.

a. Stochastic group membership: covariates map stochastically to the benefited and harmed groups. We construct a 25% minority group deterministically from covariates as before, but randomly assign p% of this group to be harmed (the remainder is benefitted). CLASH outperforms the homogeneous baseline both when p=0.5 and 0.75.

b. Smaller/larger outcome variance: smaller / larger variance in the observed outcomes. CLASH outperforms the homogeneous baseline in all settings. CLASH and the Oracle both perform better with lower variance.

c. Estimated outcome variance: variance is not known, but rather estimated from the observed outcomes. Estimating the variance has an effect on CLASH’s performance, but CLASH still outperforms the homogeneous baseline.

d. Small sample sizes: small sample sizes (N = 200, 400, 1000). CLASH outperforms the homogenous baseline by a wide margin in experiments with moderately small samples (N=1000). While the gap between CLASH and the homogenous baseline decreases as sample size decreases, CLASH still outperforms the baseline with as few as 400 participants. However, with very small samples (N=200), neither CLASH nor the homogeneous baseline stops the experiment. We have included this setting as a limitation in our revised paper.

e. Large number of covariates: high-dimensional covariates (d=100, 500). CLASH is robust to increasing dimensionality to a point, outperforming the homogeneous baseline even with 100 covariates. The extreme case with 500 covariates is more challenging: here, practitioners may need to perform feature selection before running CLASH. We discuss this in our revised limitations section.

### (2) Decision to Stop Only on Harmed Group
We now focus on what investigators can do once CLASH indicates that an experiment should be stopped. We have added this discussion to the Methods section of the revised paper.

*Stopping Decision*. When CLASH indicates a group is being harmed, investigators should make choices based on domain expertise. If the nature of harm is serious (e.g., mortality) they may decide to stop the experiment for all participants. For milder harms (e.g., crashes in an A/B test), they may decide to stop the experiment only for the harmed group. If the identified harm is much less consequential than the potential benefit (e.g., harm of increased headaches vs. benefit of curing cancer), they may decide to not stop the experiment at all. The specific choice made will depend heavily on the ethical and financial aspects of the experiment, as well as a thorough review of the interim data. CLASH is not intended to make this decision for investigators; however, it helps investigators realize that a group is being harmed and thus a stopping decision needs to be made.

*Tree-based heuristic for identifying the harmed group*: If investigators choose to stop the experiment only for harmed participants, they must first identify the harmed group. The distribution of CLASH weights at stopping time can help in this task: groups with estimated participant weights close to 1 are likely to be harmed. Fig S13 in Appendix H illustrates this in our empirical application: the estimated weights in Regions 1 and 2 are both close to 1, indicating that these are the groups on which to stop. With few covariates, investigators can manually inspect the weight distribution for each covariate combination to identify the harmed group. With many covariates, investigators can use a simple heuristic: a regression decision tree on the estimated CLASH weights can find the covariate values for which the weights are the largest. Limiting the depth of this tree can ensure that the identified group is actionable (i.e., it is possible to stop the experiment on it) and of non-trivial size.

There are alternative approaches to this harmed group identification task; for example, practitioners can use subgroup identification methods on the raw outcomes (e.g. [1]). We are agnostic to the choice of method: investigators can pick the approach most appropriate for their domain.

*Treatment effect estimation*. Stopping the experiment in only one group can affect inference at the end of the experiment, as the treatment is no longer randomly assigned across covariates. To estimate the ATE over the entire population, practitioners should use inverse propensity weights to correct for the induced selection bias. We illustrate this correction in our response to reviewer K2p1 below, and in our revised manuscript.

[1] Zhang, et al, DOI: 10.21037/atm.2018.03.07

Meta-review: This well-written paper has been assessed by 6 knowledgeable reviewers. They largely agree on accepting it: four gave straight accept, and two weak accept score. The authors provided thorough rebuttals that effectively addressed key concerns raised by the reviewers. The presented work is original and it addresses somewhat sparsely studied yet practically important problem. Considering all the above, I have no hesitation in recommending acceptance of this work, and if there is space, offering it a spot in a spotlight session.
Decision: Accept (spotlight)