SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jessica Dai
jessicadai@berkeley.edu
Paula Gradu
pgradu@berkeley.edu
Christopher Harshaw*
charshaw@mit.edu
From clinical development of cancer therapies to investigations into partisan bias, adaptive sequential designs have become increasingly popular method for causal inference, as they offer the possibility of improved precision over their non-adaptive counterparts. However, even in simple settings (e.g. two treatments) the extent to which adaptive designs can improve precision is not sufficiently well understood. In this work, we study the problem of Adaptive Neyman Allocation in a design-based potential outcomes framework, where the experimenter seeks to construct an adaptive design which is nearly as efficient as the optimal (but infeasible) non-adaptive Neyman design, which has access to all potential outcomes. Motivated by connections to online optimization, we propose Neyman Ratio and Neyman Regret as two (equivalent) performance measures of adaptive designs for this problem. We present CLIP-OGD, an adaptive design which achieves  $\widetilde{O}(\sqrt{T})$  expected Neyman regret and thereby recovers the optimal Neyman variance in large samples. Finally, we construct a conservative variance estimator which facilitates the development of asymptotically valid confidence intervals. To complement our theoretical results, we conduct simulations using data from a microeconomic experiment.
From medicine and public health to economics and public policy, randomized control trials are used in a variety of disciplines to investigate causal effects. Typically, treatment is assigned in a non-adaptive manner, where assignments are determined before any outcomes are observed. A sequential experimental approach, which adaptively assigns treatment based on previously observed outcomes, offers the possibility of more precise or high powered estimates of relevant causal effects. Adaptive experiments are run to develop clinical therapies for breast cancer [Barker et al., 2009], evaluate incentives to reduce partisan bias [Offer-Westort et al., 2021], and evaluate customer acquisition via online advertising [Schwartz et al., 2017], to name a few.
In this paper, we study the problem of Adaptive Neyman Allocation, which we informally define as follows. An optimal non-adaptive experimental design which minimizes variance of an estimator will depend on the unknown potential outcomes, rendering it infeasible to run. However, by adaptively choosing treatment assignments in a sequential manner based on observed outcomes, we can hope to guarantee that the variance of the estimator under the adaptive design converges to the optimal non-adaptive variance. The problem of Adaptive Neyman Allocation is to construct such an adaptive design which guarantees the variance converges to the (infeasible) optimal non-adaptive design.
An experimental design which sufficiently addresses the Adaptive Neyman Allocation problem offers the advantage of higher statistical power, relative to a broad class of fixed experimental designs. Practically speaking, this means that either smaller confidence intervals are obtained for a given number of experimental units, or that fewer units are required to achieve confidence intervals of a given length. In practice, this means that investigating causal effects can be cheaper—in terms of time, money, and other valuable resources—when adaptive experiments are run. Although several experimental designs have been proposed for this purpose [Hahn et al., 2011, Blackwell et al., 2022], none have provided formal guarantees that the optimal non-adaptive variance can be achieved and the effectiveness of such designs has recently been called into question Cai and Rafi [2022].
1. Neyman Ratio and Regret: We propose two (equivalent) performance measures of experimental designs for the problem of Adaptive Neyman Allocation: Neyman Ratio and Neyman Regret. We show that guarantees on the rates of these performance measures directly translate to guarantees on the convergence of variance to the Neyman variance.
2. CLIP-OGD: We propose the adaptive design CLIP-OGD, a variant of online stochastic projected gradient descent for which the Neyman regret is  $\tilde{\mathcal{O}} (\sqrt{T})$ . This guarantees that the variance of the sequential effect estimator approaches the Neyman variance.
3. Confidence Intervals: By constructing a conservative variance estimator, we provide confidence intervals which guarantee asymptotic coverage of the average treatment effect.
In Section 7, we support these theoretical results with simulations using data from a microeconomic experiment. Our results rely on viewing the Adaptive Neyman Allocation problem through the lens of online convex optimization. However, as discussed in Section 4.2, due to the subtleties arising in the problem, we do not know of an existing online algorithm which directly obtains these results.
We work within the potential outcomes framework for causal inference Neyman [1923], Rubin [1980], Imbens and Rubin [2015]. The idea of optimal treatment allocation dates back to Neyman [1934], where he demonstrates that sampling from treatments proportional to the within-treatment outcome variance will minimize the variance of standard estimators. Unfortunately, this type of design is not practically feasible when little is known about the statistics of outcomes from each treatment. Robbins [1952] highlights adaptive sampling as one of the more pressing open statistical problems at the time. In Chapter 5, Solomon and Zacks [1970] presents a survey of adaptive designs for survey sampling, but from a Bayesian perspective. More recently, Hahn et al. [2011] proposed a two stage design in a super-population setting, where data is uniformly collected from both arms in the first stage, statistics of the treatment arm are estimated, and a fixed probability derived from estimated statistics is used in the second stage. They derive the limiting distribution of the effect estimator under the two-stage design, which has a variance that is similar to, but asymptotically bounded away from the optimal Neyman variance. In a design-based setting, Blackwell et al. [2022] propose a similar two-stage approach and, through simulations, provide practical guidance on how to choose the length of the first stage. Although both of these works are motivated by achieving the Neyman variance, neither formally show that this is possible under the two-stage design.
While the goal in this paper is to increase the precision of treatment effect estimates, a variety of response-adaptive designs have been developed for various objectives, including reducing mean total sample size [Hayre and Turnbull, 1981] and reduction of harm reduction in null hypothesis testing [Rosenberger et al., 2001]. Eisele [1994] proposes the Doubly Adaptive Coin Based Design, which is a meta-algorithm for targeting various allocation proportions when outcomes are drawn i.i.d. from an exponential family. Hu and Rosenberger [2003] critiques many response-adaptive designs as being "mypoic strategies" which have "adverse effects on power", providing an asymptotic framework by which to judge adaptive design when the outcomes are i.i.d. and binary. This asymptotic evaluation framework was extended to continuous outcomes by Zhang and Rosenberger [2006]. An additional line of work has developed adaptive Bayesian methods for subgroup identification [Xu et al., 2014].
Causal inference under adaptively collected data has seen a variety of recent developments which are adjacent to, but distinct from, the problem of Adaptive Neyman Allocation. One line of research has been to construct estimators via re-weighting which ensure consistency and normality when data
is collected via bandit algorithms [Hadad et al., 2021, Zhang et al., 2020, 2021]. A second line of research has been to provide inferential methods which are valid under data-dependent stopping times [Wald, 1945, Howard et al., 2021, Ham et al., 2022]. Finally, Offer-Westort et al. [2021] propose an adaptive experimental design for improved selective inference, when only the effect of the best performing treatment is to be inferred.
The sequential experiment takes place over  $T$  rounds, where we assume that  $T$  is fixed and known to the experimenter. At each iteration  $t \in [T]$ , a new experimental unit (e.g. clinical participant), enters into the experiment, so that there are  $T$  units in total. In an abuse of notation, we identify units with their respective round  $t \in [T]$ . The experimenter assigns a (random) treatment  $Z_{t} \in \{0,1\}$  (e.g. drug or placebo) to the experimental unit. The unit has two real-valued potential outcomes  $y_{t}(1), y_{t}(0)$  which are unknown to the experimenter and represent the unit's measured response to the treatment assignments (e.g. measured heart rate). The term "potential" is used here because while only one treatment is assigned and thus only one outcome is observed, both outcomes have the potential to be observed. At the end of the round, the experimenter sees the observed outcome  $Y_{t} = \mathbf{1}[Z_{t} = 1]y_{t}(1) + \mathbf{1}[Z_{t} = 0]y_{t}(0)$ .
2.1 Potential Outcomes Framework
In this paper, we adopt a design-based framework where the sequence of potential outcomes  $\{y_{t}(1), y_{t}(0)\}_{t=1}^{T}$  is deterministic and the only source of randomness is treatment assignment itself. In particular, we place no assumption on the homogeneity of the outcomes: they are not necessarily related to each other in any systematic way. Although the potential outcomes are deterministic, we introduce finite population analogues of various statistics. Define the finite population second moments  $S(1)$  and  $S(0)$  and correlation of the treatment and control outcomes  $\rho$  to be
$$
S (1) ^ {2} = \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (1) ^ {2}, \quad S (0) ^ {2} = \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2}, \quad \text {a n d} \quad \rho = \frac {\frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (1) y _ {t} (0)}{S (1) S (0)}.
$$
Observe that the correlation between treatment and control outcomes is bounded  $\rho \in [-1,1]$ . Although we refer to  $\rho$  as the correlation, it also known as the cosine similarity and is generally not equal to the Pearson correlation coefficient. We remark that although the potential outcomes  $y_{t}(1)$  and  $y_{t}(0)$  are deterministic, the observed outcome  $Y_{t}$  is random, as it depends on random treatment assignment. The natural filtration according to these rounds is denoted as  $\mathcal{F}_1\ldots \mathcal{F}_T$ , so that  $\mathcal{F}_t$  captures all randomness before the sampling of  $Z_{t}$ , i.e. the treatments assigned and outcomes observed in previous rounds.
In this sequential setting, the mechanism for random treatment assignment can incorporate observed outcomes from previous experimental rounds. This treatment mechanism, referred to as the experimental design, is selected by and thus known to the experimenter. Formally, the experimental design is a sequence of functions  $\{\Pi_t\}_{t=1}^T$  with signature  $\Pi_t: (\{0,1\} \times \mathbb{R})^{t-1} \to [0,1]$  such that treatment is assigned as  $\operatorname*{Pr}(Z_t = 1 \mid \mathcal{F}_t) = \Pi_t(Z_1, Y_1, \ldots, Z_{t-1}, Y_{t-1})$ . We denote  $P_t = \operatorname*{Pr}(Z_t = 1 \mid \mathcal{F}_t)$  as the (random) probability of treatment assignment at iteration  $t$ , given previously observed treatment assignments and outcomes.
The causal estimand of interest is the average treatment effect, defined as
$$
\tau = \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (1) - y _ {t} (0).
$$
The average treatment effect captures the average counterfactual contrast between a unit's outcomes under the two treatment assignments. For example, this could be the average contrast of a clinical participant's heart rate under the drug or placebo. Individual treatment effects are defined as  $\tau_{t} = y_{t}(1) - y_{t}(0)$ , but they cannot be estimated without strong additional assumptions, as only one outcome is observed.
A standard estimator of the average treatment effect is the Horvitz-Thompson estimator, which weights observed outcome by the probability of their observation [Narain, 1951, Horvitz and Thompson, 1952]. For adaptive designs, the standard Horvitz-Thompson estimator is infeasible because
the marginal probability of treatment assignment  $\operatorname*{Pr}(Z_t = 1)$  depends on the unknown potential outcomes. For this reason, we investigate the adaptive Horvitz-Thompson estimator, which uses the random (observed) treatment probabilities used at each iteration.
$$
\hat {\tau} \triangleq \frac {1}{T} \sum_ {t = 1} ^ {T} Y _ {t} \Big (\frac {\mathbf {1} [ Z _ {t} = 1 ]}{P _ {t}} - \frac {\mathbf {1} [ Z _ {t} = 0 ]}{1 - P _ {t}} \Big) ,
$$
where we recall that  $P_{t} = \Pi_{t}(Z_{1},Y_{1},\ldots Z_{t - 1},Y_{t - 1})$  is the treatment probability under the experimental design given the observed data. When treatment assignments are non-adaptive and independent, then the adaptive Horvitz-Thompson estimator is the equivalent to the standard Horvitz-Thompson estimator. Such adaptively weighted estimators have been proposed previously in the literature, e.g. [Bowden and Trippa, 2015, Hadad et al., 2021]. Below, we provide positivity conditions under which the adaptive estimator is unbiased, and derive its variance.
Proposition 2.1. If  $\min \{P_t, 1 - P_t\} > 0$  almost surely for all  $t \in [T]$  then the adaptive Horvitz-Thompson estimator is unbiased:  $\mathbb{E}[\hat{\tau}] = \tau$ .
Proposition 2.2. The variance of the adaptive Horvitz-Thompson estimator is
$$
T \cdot \mathrm {V a r} (\hat {\tau}) = \frac {1}{T} \sum_ {t = 1} ^ {T} \Bigl (y _ {t} (1) ^ {2} \mathbb {E} \Bigl [ \frac {1}{P _ {t}} \Bigr ] + y _ {t} (0) ^ {2} \mathbb {E} \Bigl [ \frac {1}{1 - P _ {t}} \Bigr ] \Bigr) - \frac {1}{T} \sum_ {t = 1} ^ {T} \tau_ {t} ^ {2}.
$$
2.2 Asymptotic Framework and Assumptions
Following the convention of design-based inference, we analyze statistical methods within an asymptotic framework [see e.g., Freedman, 2008, Lin, 2013, Sävje et al., 2021]. This provides a formal basis for reasoning about the performance of statistical methods as the sample size increases, giving meaning to conventional notions such as consistency and limiting distribution. Formally speaking, the asymptotic sequence of potential outcomes is a triangular array  $\{\{y_{t,T}(1), y_{t,T}(0)\}_{t=1}^{T}\}_{T=1}^{\infty}$ , which yields a sequence of estimands  $\{\tau_T\}_{T=1}^{\infty}$  and, together with an appropriately specified sequence of experimental design, a sequence of estimators  $\{\hat{\tau}_T\}_{T=1}^{\infty}$ . Analysis which applies to a fixed  $T$  is said to be finite-sample (e.g.  $\mathbb{E}[\hat{\tau}_T] = \tau_T$ ) whereas analysis which applies to the entire sequence is said to be asymptotic (e.g.  $\tau_T - \hat{\tau}_T \xrightarrow{p} 0$ ). Although we use an asymptotic framework, we emphasize that the majority of our results are derived from finite-sample analysis and are merely interpreted through the lens of the asymptotic framework. We drop the subscript  $T$  for notational clarity.
The main regularity conditions we place on the sequence of potential outcomes is below.
Assumption 1. There exist constants  $0 < c \leqslant C$  with  $c < 1$  such that for all  $T$  in the sequence:
1. Bounded Moments:  $c \leqslant \left( \frac{1}{T} \sum_{t=1}^{T} y_t(k)^2 \right)^{1/2} \leqslant \left( \frac{1}{T} \sum_{t=1}^{T} y_t(k)^4 \right)^{1/4} \leqslant C \forall k \in \{0, 1\}$ .
2. Bounded Correlation:  $\rho \geqslant -(1 - c)$
The upper moment bound in Assumption 1 stipulates that the potential outcomes cannot grow too large with the sample size, while the lower moment bound is a type of non-degeneracy condition that prevents an increasingly large fraction of the outcomes going to zero. These assumptions are analogous to finite fourth moment and positive second moment assumptions in an i.i.d. setting. The bounded correlation assumption stipulates that the treatment and control outcomes are not exactly negatively correlated. In this paper, we do not assume that these constants  $C$  and  $c$  are known to the experimenter; however, if the experimenter can correctly specify such bounds (perhaps knowing a priori the scaling of the outcomes) then some of the constant factors in our analysis can be improved. We emphasize here that Assumption 1 places no assumption on the order in which units arrive in the experiment. In this sense, Assumption 1 allows for arbitrary "non-stationarity" or "drift" in the potential outcomes over the experimental rounds. In the next section, these regularity assumptions will ensure that the Neyman variance converges to zero at the parametric rate.
3 Neyman Design: The Infeasible Non-Adaptive Ideal
The problem of Adaptive Neyman Allocation is to construct an adaptive experimental design that achieves nearly the same variance as an optimal non-adaptive experimental design, chosen with
knowledge of all potential outcomes. The optimal non-adaptive design, referred to as the Neyman Design, is infeasible to implement because it depends on all potential outcomes, which are unknown to the experimenter at the design stage. The goal is that an adaptive experimental design—which can select treatment assignment based on observed outcomes—can gather enough information to perform as well as the infeasible Neyman design.
In order to define the optimal non-adaptive design, we begin by defining the class of Bernoulli designs. Informally, the class of Bernoulli designs consists of non-adaptive designs where each unit receives treatment  $Z_{t} = 1$  with probability  $p$ , independently of past treatment assignments and observations. Formally, this class is parameterized by a non-adaptive sampling probability  $p \in [0,1]$  such that for all  $t \in [T]$ , the treatment policy  $\Pi_t$  is a constant function whose value is  $p$ . Using Proposition 2.2, we can derive the variance of the Bernoulli design with parameter  $p \in [0,1]$  to be
$$
T \cdot V _ {p} = S (1) ^ {2} \left(\frac {1}{p} - 1\right) + S (0) ^ {2} \left(\frac {1}{1 - p} - 1\right) + 2 \rho S (1) S (0).
$$
From the above, we can see that in order to minimize the variance of the Horvitz-Thompson estimator under the Bernoulli design, we should set the sampling probability  $p$  so as to balance the square of the second moments of treatment and control outcomes. The Neyman Design is the Bernoulli design which minimizes the variance of the Horvitz-Thompson estimator. The corresponding optimal probability  $p^*$  and variance  $V_{\mathrm{N}}$  are referred to as the Neyman probability and Neyman variance, respectively. The following proposition derives these quantities in terms of the potential outcomes.
Proposition 3.1. The Neyman variance is  $T \cdot V_{N} = 2(1 + \rho)S(1)S(0)$ , which is achieved by the Neyman probability  $p^{*} = (1 + S(0) / S(1))^{-1}$ .
In order to quantify the reduction in variance achieved by the Neyman design, define the relative Neyman efficiency with respect to  $p \in [0,1]$  to be  $V_{\mathrm{N}} / V_{p}$ . Intuitively, this ratio is a scale-free measure which captures the percent reduction in variance of the sequential Horvitz-Thompson estimator under the Neyman design. Formally, the equation for the relative Neyman efficiency is given below:
$$
\frac {V _ {\mathrm {N}}}{V _ {p}} = 2 (1 + \rho) \left[ \frac {S (1)}{S (0)} \cdot \frac {(1 - p)}{p} + \frac {S (0)}{S (1)} \cdot \frac {p}{(1 - p)} + \rho \right] ^ {- 1}.
$$
Consider the setting where outcomes are uncorrelated, and treatment outcomes are larger than control outcomes, e.g.  $\rho = 0$ ,  $S(1) = 4 \cdot S(0)$ . In this case, the Neyman design is able to achieve less than half the variance of the uniform Bernoulli design (with  $p = 1/2$ ): we can plug in to 3 to see that in this setting, we have  $V_{N} / V_{p} = 0.47$ . The improvement is larger if the experimenter makes erroneous assumptions about the relative magnitudes of the treatment and control outcomes and attempts to set  $p$  accordingly: for example, if the experimenter had set  $p = 1/4$ , incorrectly believing that  $S(1) \leqslant S(0)$ , then the Neyman allocation results in a sixfold improvement in variance. Blackwell et al. [2022] derives qualitatively similar analysis of Neyman efficiency for stratified designs.
While the relative Neyman efficiency is helpful in determining the variance reduction afforded by the (infeasible) optimal Bernoulli design, it does not address the main question: which adaptive experimental designs can guarantee similar variance reduction? In the next section, we propose a performance metric which better addresses this question.
4 Adaptive Neyman Allocation: An Online Optimization Approach
4.1 Neyman Ratio and Neyman Regret: New Performance Measures
Let  $V$  be the variance of the adaptive experimental design. We introduce our first performance measure of a sequential experimental design for Adaptive Neyman Allocation.
Definition 1. The Neyman ratio of a sequential experimental design is  $\kappa_{T} = (V - V_{\mathrm{N}}) / V_{\mathrm{N}}$
The subscript  $T$  in  $\kappa_{T}$  in included the reflect dependence of the number of rounds  $T$ . The Neyman ratio is motivated by the following relationship between the adaptive variance and the optimal Neyman variance:
$$
V = \left(\frac {V}{V _ {\mathrm {N}}}\right) \cdot V _ {\mathrm {N}} = \left(1 + \kappa_ {T}\right) \cdot V _ {\mathrm {N}}. \tag {1}
$$
Equation (1) shows that the adaptive design can recover the Neyman variance if and only if the Neyman ratio  $\kappa_{T}$  can be made arbitrarily small. For this reason, we propose the Neyman ratio as a performance measure of a sequential experimental design.
A natural question then becomes: how small can the Neyman ratio  $\kappa_{T}$  be made as the number of rounds  $T$  increases? To answer this question, we view the problem of minimizing the Neyman ratio through the lens of online optimization. To this end, we must re-express the variance of the sequential experimental design. For each round  $t\in [T]$ , define the cost function  $f_{t}:[0,1]\to \mathbb{R}$  as  $f_{t}(p) = y_{t}(1)^{2} / p + y_{t}(0)^{2} / (1 - p)$ . Observe that by Proposition 2.2, the variance is given by  $T\cdot \mathrm{Var}(\hat{\tau}) = \mathbb{E}\big[\frac{1}{T}\sum_{t = 1}^{T}f_{t}(P_{t})\big]$ . This reformulation of variance does not allow us to minimize variance directly, for the usual reason that the outcomes, and thus the cost functions  $f_{t}$ , are not fully observed. On the other hand, our goal is only to show that the variance of the adaptive design is comparable to the Neyman variance.
Definition 2. The Neyman regret of a sequential experimental design is
$$
\mathcal {R} _ {T} = \sum_ {t = 1} ^ {T} f _ {t} (P _ {t}) - \min  _ {p \in [ 0, 1 ]} \sum_ {t = 1} ^ {T} f _ {t} (p) .
$$
Recall that  $P_{t}$  is the random treatment probability at round  $t$ . The Neyman regret compares the accumulated costs  $f_{t}(P_{t})$  incurred by the adaptive design to the accumulated costs incurred by the optimal Bernoulli design which has access to all potential outcomes. The Neyman regret is random because the sequence  $P_{1},\ldots P_{T}$  is random. The following theorem connects the expected Neyman regret to the Neyman ratio.
Theorem 4.1. Under Assumption 1, the Neyman ratio is within a constant factor of the  $1 / T$ -scaled expected Neyman regret:  $\kappa_T = \Theta \left(\frac{1}{T}\mathbb{E}[\mathcal{R}_T]\right)$ .
Theorem 4.1 demonstrates that the Neyman ratio can be made small by minimizing the expected Neyman regret in an online fashion. In particular, any sublinear bound on the expected Neyman regret ensures that the Neyman ratio goes to zero so that, in large samples, the adaptive design achieves the variance reduction of the optimal Neyman design. Any adaptive design which aims to achieve Neyman variance must, to some extent, minimize expected Neyman regret.