SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hammaad Adam*  
Massachusetts Institute of Technology
Fan Yin Microsoft Corporation
Huibin (Mary) Hu  
Microsoft Corporation
Neil Tenenholtz  
Microsoft Research
Lorin Crawford  
Microsoft Research
Lester Mackey  
Microsoft Research
Allison Koenecke Cornell University
Randomized experiments often need to be stopped prematurely due to the treatment having an unintended harmful effect. Existing methods that determine when to stop an experiment early are typically applied to the data in aggregate and do not account for treatment effect heterogeneity. In this paper, we study the early stopping of experiments for harm on heterogeneous populations. We first establish that current methods often fail to stop experiments when the treatment harms a minority group of participants. We then use causal machine learning to develop FLASH, the first broadly-applicable method for heterogeneous early stopping. We demonstrate FLASH's performance on simulated and real data and show that it yields effective early stopping for both clinical trials and A/B tests.
Randomized experiments are the gold-standard method of determining causal effects, whether in clinical trials to evaluate medical treatments or in A/B tests to evaluate online product offerings. The sample size and duration of such experiments are typically specified in advance. However, there are often strong ethical and financial reasons to stop an experiment before its scheduled end, especially if the treatment shows early evidence of harm [7]. For example, if early data from a clinical trial demonstrates that the treatment increases the rate of serious side effects, it may be necessary to stop the trial to protect experimental participants [9].
A variety of statistical methods can be used to determine when to stop an experiment for harm [22, 39, 14, 13]. Investigators in both clinical trials and A/B tests will often choose to use a subset of these methods—collectively referred to as “stopping tests”—based on the specifics of their domain. Stopping tests not only identify harmful effects from early data, but also limit the probability of stopping early when the treatment is not harmful. However, stopping tests are typically applied to the data in aggregate (i.e., “homogeneous”) and do not account for heterogeneous populations. For example, a drug may be safe for younger patients but harm patients over the age of 65. While a growing body of literature has studied how to infer such heterogeneous effects [see, e.g., 41], little prior research has described how to adapt stopping tests to respond to heterogeneity.
We continue the above example to illustrate why stopping tests should account for heterogeneity. Consider a clinical trial for a drug such as warfarin, which has no harmful effect on the majority of the population but increases the rate of adverse effects in elderly patients [32]. Using a simple simulation, we demonstrate that if elderly patients comprise  $10\%$  of the trial population, then applying a stopping
*Corresponding author (hadam@mit.edu).
†Current affiliation: Amazon. Work performed while at Microsoft (prior to joining Amazon).
Figure 1: Stopping tests applied homogeneously may not stop trials in which the treatment harms only a minority group of participants. For example, if the treatment harms  $10\%$  of all participants, the trial will stop less than  $20\%$  of the time, even if the harmful treatment effect is strong (boxed in red). We plot the stopping probability of a commonly used stopping test (O'Brien-Fleming) in situations where the treatment harms the minority group but has no effect on the majority group.
test homogeneously would rarely stop the trial for harm (Fig. 1). In most cases, the trial continues to recruit elderly patients until its scheduled end, many of whom will be harmed by their participation. This outcome violates the bioethical principle of non-maleficence [37] and is clearly undesirable.
In this paper, we consider early stopping for harm with heterogeneous populations. We first formalize the problem of heterogeneous stopping and establish the shortcomings of the prevailing homogeneous approach. We then present our novel method: Causal Latent Analysis for Stopping Heterogeneously (CLASH). CLASH uses causal machine learning to infer the probability that a participant is harmed by the treatment, then adapts existing stopping tests to better detect this harm. CLASH allows an investigator to use their stopping test of choice: it is thus flexible and easy-to-use. We theoretically establish that, for sufficiently large samples, CLASH is more likely to stop a trial than the homogeneous approach if the treatment harms only a subset of trial participants. CLASH also does not stop trials unless a group is harmed; it thus increases statistical power while limiting Type I error rate. We demonstrate these properties in an extensive series of experiments on synthetic and real data, and establish that CLASH leads to effective heterogeneous stopping across a wide range of domains.
While a small number of papers have attempted to address the problem of heterogeneous early stopping, they have relied on restrictive assumptions, limiting their applicability. Thall and Wathen [35] require knowledge of the source of heterogeneity, which is rarely available in practice. Yu et al. [42] allow for unknown groups but only model linear heterogeneity. Yu et al. [43] relax the linearity restriction but do not handle time-to-event data commonly observed in clinical trials. In contrast, FLASH does not require prior knowledge, makes no parametric assumptions, and works with any data distribution. It is thus the first broadly applicable tool for early stopping with any number of heterogeneous groups.
We emphasize that early stopping is a nuanced decision. For example, if a treatment harms only a subset of participants, it may be desirable to stop the experiment only on the harmed group but continue it on the rest of the population. In other situations, it may make sense to stop the trial altogether. The decision to stop a trial to protect one group may disadvantage another group that would have benefited from the treatment; any decision on early stopping must thus carefully weigh the treatment's potential benefit, the size of the harmed group, the nature of the harm, and other ethical considerations. Our method is not intended to replace such discussion. It can, however, serve as an aid for investigators to make difficult decisions on early stopping.
2.1 Randomized Experiment
Consider an experiment that evaluates the effect of a binary treatment  $D \in \{0,1\}$  on an observed outcome  $Y$ .  $D$  is assigned uniformly at random to participants in the experiment.  $Y$  is chosen specifically to measure harm and may be different from the experiment's primary outcome of interest. For example, in a clinical trial,  $Y$  could convey a primary outcome such as mortality or a secondary outcome such as the occurrence of serious side effects. We allow for both scenarios, as experiments
often care about harm on multiple dimensions. Assume that an increase in  $Y$  is harmful. The harm caused by the treatment can be measured by the average treatment effect (ATE) of  $D$  on  $Y$ ,  $\mathrm{ATE} = \mathbb{E}[Y|D = 1] - \mathbb{E}[Y|D = 0]$ . A positive ATE implies that the treatment is harmful, while a negative ATE implies that it is beneficial. Note that the ATE has a natural estimator: the difference in the observed means of  $Y$  between the treated and untreated groups.
Now, it is not necessary that every individual in the population responds to the treatment in the same way. The treatment may harm some individuals, benefit some, and have no effect on others. Assume that the population can be divided into  $M$  discrete groups, where random variable  $G$  conveys group membership. For group  $g$ , we define the conditional average treatment effect (CATE)  $\tau(g) = \mathbb{E}[Y|D = 1, G = g] - \mathbb{E}[Y|D = 0, G = g]$ . Note that in general, we do not observe  $G$ . Rather, we typically observe a set of covariates  $X$  that correlate with group membership. We define the CATE for a participant with covariates  $x$  as  $\tau(x) = \mathbb{E}_{g|x}[\tau(g)]$ .
The experiment can recruit a maximum of  $N$  participants and is run for  $T$  time steps, where the values of  $N$  and  $T$  are set at the experiment's start. For simplicity, assume that we recruit two participants at each time step—one in treatment and one in control—and so  $T = N / 2$ . For each participant  $i$ , we measure treatment assignment  $d_{i}$ , covariates  $x_{i}$ , and observed outcome  $y_{i}$ , but do not observe group membership  $g_{i}$ . The expected effect of the treatment on participant  $i$  is given by  $\tau(g_{i})$ . We assume that  $(d_{1}, x_{1}, y_{1}, g_{1}), \ldots, (d_{N}, x_{N}, y_{N}, g_{N})$  are independent and identically distributed.
2.2 Early Stopping for Harm
If the treatment is harmful, it is often important to end the experiment as early as possible, to minimize harm for participants and financial loss for the experimenter. Clinical trials and A/B tests usually build in interim checkpoints, where the data collected thus far is evaluated to make a decision on whether to continue the experiment. However, it is also important to not stop the experiment at these checkpoints unless the treatment is actually harmful; incorrectly stopping early could result in lost innovation or societal benefit. Achieving this balance requires using dedicated statistical methods for early stopping, collectively known as "stopping tests."
A large body of literature has studied stopping tests. Frequentist methods like the Pocock adjustment [23], O'Brien-Fleming (OF) adjustment [22], and alpha-spending [10] are common in clinical trials with a pre-specified schedule of checkpoints. True sequential methods that allow for continuous monitoring are often based around Wald's Sequential Probability Ratio Test (SPRT) [39], such as the 2-SPRT [19], MaxSPRT [16], and mixture-SPRT (mSPRT) [26, 14]. Other sequential methods are based on martingales [3, 25] and testing by betting [40]. Such tests are more practical in A/B testing scenarios, where investigators can monitor responses as they arrive. There is also a growing interest in Bayesian designs and stopping tests, both for clinical trials [4] and A/B tests [33]. App. A summarizes some commonly-used stopping tests.
While these stopping tests differ in their details, many have the same general form: compute a test statistic from the hitherto observed data  $\lambda_{n}(y_{1:n},d_{1:n})$ , then stop the experiment if this test statistic exceeds some bound  $b(\alpha)$ . Here,  $n$  is the number of outcomes observed thus far and  $\alpha$  is the desired bound on the rate of unnecessary stopping (i.e., stopping when the treatment is not harmful). For clarity, we provide a concrete example of a stopping test: the OF-adjusted z-test. Assume that  $Y\sim \mathcal{N}(D\mu ,\sigma^2)$ , where  $\sigma^2$  is known but  $\mu$  is not. The treatment effect is homogeneous, and so the ATE (i.e.,  $\mu$ ) conveys if the treatment is harmful. To test between a null hypothesis of no harmful effect  $(H_0:\mathrm{ATE}\leq 0)$  and an alternate hypothesis of a harmful effect  $(H_{1}:\mathrm{ATE} > 0)$ , we use
$$
\lambda_ {n} ^ {O F} = \frac {\sqrt {n}}{\sqrt {2 \sigma^ {2}}} \left(\frac {\sum_ {i = 1} ^ {n} y _ {i} d _ {i}}{n / 2} - \frac {\sum_ {i = 1} ^ {n} y _ {i} (1 - d _ {i})}{n / 2}\right). \tag {1}
$$
The OF-adjusted z-test compares this test statistic to an appropriate bound  $b^{OF}(\alpha)$ , which depends both on the number and timing of checkpoints. For example, if we were to conduct one checkpoint halfway through the trial, then  $b^{OF}(0.05) = 2.37$ . If  $\lambda_{N/2}^{OF} > b^{OF}$ , the test stops the experiment for harm; else, the experiment continues. Using the OF adjustment guarantees that if the treatment is not harmful (i.e., ATE  $\leq 0$ ), then the probability of stopping the experiment is at most  $5\%$ .
2.3 Stopping with Heterogeneous Populations
Stopping tests are typically applied to the data in aggregate and define harm in terms of the ATE. While this approach is reasonable if the treatment effect is homogeneous, there is no guarantee that it
will stop the experiment if the treatment only harms a subset of participants. For example, consider a situation in which there are two equally sized groups with equal but opposite treatment effects, that is,  $p(G = 0) = p(G = 1)$  and  $\tau(0) = -\tau(1)$ . The ATE with this mixture distribution is zero, and so any stopping test with  $H_0: \mathrm{ATE} \leq 0$  is designed to continue to completion at least  $(1 - \alpha)\%$  of the time. Unfortunately, the failure to stop means that half of the trial participants will be harmed.
Experiments with heterogeneous populations thus require a more precise definition of harm. To comply with the bioethical principle of non-maleficence, experiments should be stopped not only if the treatment is harmful in aggregate (i.e., ATE  $>0$ ), but also if the treatment harms any group of participants (i.e.,  $\exists g$  s.t.  $\tau(g) > 0$ ). We define the null hypothesis of no group harm,
$$
H _ {0}: \tau (g) \leq 0 \forall g \in \{1, \dots , M \}. \tag {2}
$$
An effective stopping test must fulfill two conditions. First, it must have high stopping probability if the treatment harms any group of participants. Second, it must limit the probability of stopping if no group of participants is harmed (i.e., limit the "rate of unnecessary stopping"). Note that these conditions are equivalent to high statistical power and low Type I error rate relative to (2). For the rest of the paper, we use these two metrics—the stopping probability if any group is harmed and the rate of unnecessary stopping—to evaluate stopping tests with heterogeneous populations.
Fig. 1 demonstrates the limitations of applying stopping tests homogeneously to heterogeneous populations. We simulate an experiment that runs for 1,000 time steps and recruits 2,000 participants. Participants come from two groups, with  $G \in \{0,1\}$  indicating membership in a minority group. The treatment has no effect on the majority group but harms the minority with treatment effect  $\theta$ , with  $Y \sim \mathcal{N}(\theta \times D \times G, \sigma^2)$  and  $\sigma^2 = 1$ . The trial has one checkpoint at the halfway stage and uses an OF-adjusted z-test to decide whether to stop for harm. We vary  $\theta$  from 0 to 1 and  $\mathbb{P}(G = 1)$  from 0.1 to 0.5, and plot the stopping probability across 1,000 replications. We find that as the size of the harmed group decreases, stopping probability falls significantly. Continuing our warfarin example from Sec. 1, if elderly patients comprise  $10\%$  of trial participants (lowest line in Fig. 1), then the stopping probability is less than  $20\%$ , even if the treatment has a large harmful effect  $(\theta = 1)$ .
3 Causal Latent Analysis for Stopping Heterogeneously (CLASH)
Thus far, we have introduced early stopping for harm and established that existing methods applied homogeneously may not be effective when applied to heterogeneous populations. In this section, we first develop useful intuition on heterogeneous early stopping. We then present CLASH, our new two-stage approach. Finally, we theoretically establish that CLASH improves stopping probability if the treatment harms any group of participants without inflating the rate of unnecessary stopping.
The key problem with the homogeneous approach is that it averages over potential heterogeneous groups. One solution to this problem is to run a stopping test separately on different subsets of the population, but a question remains: which subset guarantees a high stopping probability? Assume for a moment that we have prior knowledge of each participant's group  $g_{i}$  and their CATE  $\tau(g_{i})$ . Then, Prop. 3.1 (proved in App. E.1) establishes that for  $n$  sufficiently large, the stopping test that only considers participants from harmed groups stops with probability approaching 1.
Proposition 3.1 (Group knowledge improves early stopping). Let  $\mu^{*} = \mathbb{E}[\tau(g_{1})|\tau(g_{1}) > 0]$  denote the ATE for harmed groups and  $S_{n}^{*} = \{i \leq n : \tau(g_{i}) > 0\}$  the participants from harmed groups. Let  $\lambda_{n}^{*} = \Delta_{n}^{*} / \hat{\sigma}_{n}^{*}$  denote the value of a test statistic computed only on  $S_{n}^{*}$ , where  $\mathbb{E}\Delta_{n}^{*} = \mu^{*}$ ,  $(\Delta_{n}^{*} - \mu^{*}) / \hat{\sigma}_{n}^{*} \stackrel{d}{\to} \mathcal{N}(0,1)$ ,  $\sqrt{|S_{n}^{*}|}\hat{\sigma}_{n}^{*} \stackrel{p}{\to} \sigma^{*}$ , and  $|S_{n}^{*}|$  diverges in probability to infinity. Fix any stopping threshold  $b(\alpha)$ . Then, the stopping test that only considers  $S_{n}^{*}$  has stopping probability converging to 1, i.e.,  $\mathbb{P}(\lambda_{n}^{*} > b(\alpha)) \to 1$ .
The assumptions of Prop. 3.1 hold for many popular frequentist hypothesis tests, including an OF-adjusted z-test, an OF-adjusted t-test, and an OF-adjusted stopping test to detect differences in restricted mean survival time (RMST) for time-to-event data [28]. To give a rough intuition of
<sup>3</sup>We do not refer to these as statistical power and Type I error rate to prevent confusion between the hypothesis test of the overall experiment and that of the stopping test. However, we note the equivalence.
the proof, the assumed conditions imply that the mean of  $\lambda_{n}^{*}$  grows with  $|S_{n}^{*}|$  while its variance remains bounded, and so  $\lambda_{n}^{*}$  eventually exceeds any fixed bound. Prop. 3.1 establishes that to stop the experiment with high probability, the stopping test should be run only on the set of harmed groups  $S_{n}^{*}$ . We emphasize that the same stopping test applied to the whole population has no such guarantees. For example, if the overall ATE is 0, then the stopping probability of the test applied homogeneously is  $\leq \alpha$ . Thus, running the stopping test only on  $S_{n}^{*}$  stops the trial with far higher probability.
One way to apply a stopping test only to the harmed groups is to assign a weight to each observation  $w_{i}^{*} = \mathbb{1}[i\in S_{n}^{*}]$ , then reweight the test statistic by this indicator. For example, for the OF-adjusted z-test in (1), using the following weighted test statistic is equivalent to running the test only on  $S_{n}^{*}$
$$
\lambda_ {n} ^ {*} O F = \frac {\sqrt {\sum_ {1 = 1} ^ {n} w _ {i} ^ {*}}}{\sqrt {2 \sigma^ {2}}} \left(\frac {\sum_ {i = 1} ^ {n} w _ {i} ^ {*} y _ {i} d _ {i}}{\sum_ {1 = 1} ^ {n} w _ {i} ^ {*} d _ {i}} - \frac {\sum_ {i = 1} ^ {n} w _ {i} ^ {*} y _ {i} (1 - d _ {i})}{\sum_ {1 = 1} ^ {n} w _ {i} ^ {*} (1 - d _ {i})}\right). \tag {3}
$$
Prop. 3.1 implies that using  $\lambda_{n}^{*OF}$  leads to high stopping probability. We thus call  $w_{i}^{*}$  the optimal weights, as using them to reweight  $\lambda_{n}$  guarantees stopping probability close to 1 for large enough  $n$ .
Of course, this insight is not of direct practical use, as it assumes knowledge of  $\tau(g_i)$  and  $g_i$ , neither of which is observed. However, it motivates how to construct better stopping tests for heterogeneous populations. If we can infer from the data which groups are harmed, then we can use this information to accelerate early stopping. Specifically, if we can estimate the optimal weights  $w_i^*$ , then we can use these estimates to construct weighted tests that have high stopping probability. In contrast, stopping tests applied homogeneously will have no such guarantees. This is the insight that drives our method CLASH, which we present in the following section.
3.2 Method
At any interim checkpoint  $n$ , CLASH operates in two stages. In Stage 1, CLASH uses causal machine learning to estimate the probability that each participant  $i$  is harmed by the treatment. Then in Stage 2, it uses these inferred probabilities to reweight the test statistic of any chosen stopping test.
Stage 1: Estimate  $w_{i}^{*}$  We first estimate the treatment effect  $\tau (x_{i})$  for each participant  $i\leq n$  using any method of heterogeneous treatment effect estimation. The specific choice of method is left to the investigator; potential options include linear models, causal forests [38], and meta-learners [17].
To maintain conditional independence between the estimated CATE and the observed outcome for participant  $i$ , we exclude  $(x_{i},y_{i})$  from the training set when estimating  $\tau (x_{i})$ . We thus use  $\hat{\tau}_{n, - i}(x_i)$  to denote the estimated CATE for participant  $i$ , and  $\hat{\sigma}_{n, - i}(x_i)$  for the estimated standard error of  $\hat{\tau}_{n, - i}(x_i)$  (further discussed below). While the notation suggests leave-one-out cross validation, the estimation can instead use k-fold or progressive cross-validation [5]. We then set
$$
\hat {w} _ {i} ^ {n} = 1 - \Phi \left(\frac {\delta - \hat {\tau} _ {n , - i} \left(x _ {i}\right)}{\hat {\sigma} _ {n , - i} \left(x _ {i}\right)}\right) \tag {4}
$$
where  $\Phi$  is the cumulative distribution function (CDF) of the standard normal and the hyperparameter  $\delta > 0$  (discussed below) is a small number. Intuitively,  $\hat{w}_i^n$  conveys the probability that  $i$  was harmed by the treatment; we establish that it is a good estimate of the optimal  $w_i^*$  in Sec. 3.3. We also discuss the choice of  $\Phi$  in Sec. 3.3.
Stage 2: Weighted Early Stopping We now use  $\hat{w}_i^n$  to weight the test statistic of any existing stopping test. The choice of test is left to investigators based on their experiment and domain. For example, CLASH with the OF-adjusted z-test would use the test statistic from (3), replacing  $w_i^*$  with  $\hat{w}_i^n$ . We provide examples of weighted test statistics for other stopping tests in App. C.
We make three important practical notes. First, cross-validation (CV) in Stage 1 is employed to prevent unnecessary stopping with small samples. CV ensures that if the treatment has no effect, then the predicted CATE for participant  $i$  will be independent of its observed outcome. Without this independence, outcomes that are large by random chance could be assigned high weights, especially in small samples. Second, most CATE estimation methods are designed to correct for confounding, which is not required in the experimental setting since the treatment is randomly administered. This simplifies the estimation problem (see App. D). Third, the FLASH weights (4) require not only point estimates of the CATE for each participant, but also standard errors. While bootstrapping can be used to obtain standard errors for any CATE estimation method, it is often computationally expensive. To save time and resources, investigators can use methods that compute standard errors during the process of fitting (e.g., causal forests or linear models).
Finally, we briefly discuss how to set the hyperparameter  $\delta$ . For our theoretical guarantees to hold (Sec. 3.3),  $\delta$  must be set to a number smaller than the smallest harmful treatment effect. This choice ensures that  $\hat{w}_i^n$  converges to  $w_i^*$  for all values of  $\tau(x_i)$ . In practice, we find that it is sufficient to set  $\delta$  to a number that reflects the experiment's minimum effect size of interest. It is standard for clinical trials and A/B tests to specify this effect size, especially for pre-hoc power calculations [34]. Thus,  $\delta$  can be set using the standard experimental procedures.